Jenkins integration - request for input
Dear testers it is well known that Jenkins is used widely to automate builds and some test activities, even though it is not explicitly designed to support test automation execution. People write t...
Forum Discussion
A few things comes to mind:
Some things not sure if these should be implemented by the plugin or if this is a TestComplete project setup or other scripting:
Reporting the result must be displayable to Jenkins. The output that TestExecute/TestComplete now creates is not easy to get into a good way in Jenkins. When an error occurs the details of the error must be displayable in Jenkins preferable with a screenshot.
Trend reports. Jenkins should able to see the test trend just like unit tests. How many tests/scenarios are there.
The executable must return proper status code so that Jenkins can interpret the result.
Some things not sure if these should be implemented by the plugin or if this is a TestComplete project setup or other scripting:
We typically run different kind of tests in different stages in the build. A typical smoke tests which are run on every commit/push then a more thorough test sweet as the build is moved between promotion levels in Jenkins. I think this is something many uses and an Jenkins integration should at least document how this kind of scenario should be handles with TestExecute/TestComplete.
Initialization of environment. With functional tests there is a need to initialize the environment with test data. This is a problem in any test execution scenario and I'm not sure that a plugin is the right place to handle this, but at least the preferred pratice(s) regarding this problem should be documented.