Team
Our general setup in our scrum team was this:
- 4 developers
- 1 tester (spending ~70% of his time for this team for general testing purposes)
- 1 product manager (spending ~60% of his time for this team)
Now we are
- 7 developers
- 1 tester (spending ~70% of his time for this team for general testing purposes)
- 1 product manager (spending ~60% of his time for this team)
- 1 student working ~30% for testing purposes
I am the tester above, having studied computer science, but I do not have any experience working as a developer.
Process
In our team I was the only tester, responsible for testing in general. Before I started there was no automated testing at all and developers had the attitude that developing software is just writing code that is writting as quick as possible and did not take responsibility for any testing purpose.
My believe is, that creating good code could not be done without various testing-activities including automated regression testing, code reviews,... Nobody wants to (or could do) do regression testing all the time so it is a must to automate this.
So I started step by step introducing more and more of these activities. I introduced them in this order:
- automated GUI regression tests (TestComplete and TestExecute)
- code reviews
- integration tests
- unit tests
It would have been better to do it in a different order but it was not possible. It have been hard fights with the head of development but the product manager was on my side and we could convince our CEOs as well. (some major bugs helped 🙂 )
We were able to make the developers take responsibility for manual testing, code reviews, unit tests and integration tests.
Now the last step is coming that the developers also should do the automated regression testing.
Right now I am doing a bit of automation but mainly focus on improving the process (teaching one of the developers checking daily the logs of our tests) and improving the process and testenvironment of the regression test (testing architect). We have a computer science student working ~1.5 days per week automating issues with TC. (we had another computer science student working for us the last year. Most of the time she worked 1.5 days per week and half a year she worked full time)
Regression testsuite (TC, TE)
Our testing suite in TestComplete is huge.
We track coverage of what to automate and what not in all of our JIRA-issues. Everything that can somehow be seen in the UI has to be covered in this testing suite.
If the suite would be run completely I think it would take ~11-12 hours. As this is too slow to get results in time we split the tests and run them in parallel. Unfortunately this could not be done properly in TestComplete itself. So we wrote some scripts doing this for us. Now the tests are automatically started each night (or manually during the day) by a job in jenkings. Afterwards the test runs on 7 virtual machines in parallel providing the result within ~2 hours. So after 2 hours our complete software is regression tested.
The test is under version control (git) and before the start of the test the test is checkout out.
All of our testitems can be run on their own. No testitem depends on another. The testitems should take less than 4-5 minutes (there are some legacy tests, that run stable but take way too long).
We are mapping the needed items for test before writing tests using them.We only map manually! The namemapping is the core of the test. Without a good and clean namemapping the test is not possible.
We extensively use extended find (if possible for all elements).
We do not map elements by name or stuff but by IDs. The developer are responsible for providing those IDs and it is a bug if they do not.
We remove all elements (panel, cell, table,...) from the namemapping that have no ID.
This way the test could be read by someone not familar with TC but knowing the software under test.
We use keyword tests but for all regularly needed stuff we have scripts implemented, like restoring databases, logging in, checking logs, comparing excel files, working with SVG-elements, event handlers, ...
We also make scripts outside TestComplete relocate the log folder dynamically on some central huge harddrive so that logs are grouped by software version tested, so we do not have to collect them from the 7 machines. Other scripts also rename the logname and logfolder so that we could directly recognize which version tested, which part of the test and how many bugs.
The test also makes an entry in a database when it is run (majorVersion, branch, revision, timestamp, testpart, number of errors,...).
A website gathering data from this database gives an overview for all builds if they are tested or not, if the test is OK or NOT OK and if the software was deployed on any of our systems (the OK, NOK naturally has to be set manually).
Our next task is make the test run in various browsers (mainly Edge). Right now the test just runs in Chrome.
But we already have scripts written that switch the region-folder for each project. (as there is no possibility to do this in TestComplete itself)
The first testrun showed that ~80% of the test runs properly in Edge, thanks to the good namemapping. Just some elements have to be changed slightly.
