Jenkins integration - request for input

Just an opinion, but wouldn't time be better served writing a proper API that ANY build/test management tool can interface with? The main problem I have with automating TC is the oudated, rigid, and oversimplified API. It's a pain in the butt to get working properly, and when you do get it to work properly you have missing features at every corner. Stuff like accessing project or project suite variables, or pausing the running test, or running a specific test group in different configs, etc, are all impossible to do through COM right now, today. Write a PROPER managed library to interface with TestComplete, and all those problems go away. Then you can use your own library to write plugins to all those other softwares without hassle.



I just don't see how a single plugin helps the situation at all. Now you have yet another non-unified piece of code to support and maintain in the already huge mess that must be TestComplete's source code. If you had a proper API, most of the other vendors (JetBrains, Jenkins, etc) would already have written their own plugin for TestComplete, that you wouldn't even have to support.



My rant is related to the fact that I tried (and utterly failed) to integrate TC into TeamCity months ago, and the programming model for automating TestComplete left me with a very sour taste.



TestComplete does its job well, but honestly when you dig past the skin layer it reeks of bad programming practices and archaic software design. You should fix THAT before trying to add new features only a small percentage of your users will ever have a need for.

As an example of what would be truly useful for programmers and indispensable to properly integrate your product into any other software out there, I wish I had a managed object oriented library that could open up a test suite, manipulate ANY of its multiple aspects (project properties, variables, actual results data, not a HTML rendition of it, breakpoints, everything else...) then start up testcomplete or testexecute through COM without loading any UI crap that makes it consume half the memory on the VM (like in current state of affairs where you just hide the main window; the only thing that saves you is paint events...). The interop should be talking directly to the test engine, not to the UI as it is doing now. Expose everything in a nice class hierarchy, with collections and events where appropriate. If I had that, I could integrate TC into anything. Heck I could design my own IDE around it. It solves both our problems and makes anyone who wants to integrate testcomplete into anything happy forever after.

A few things comes to mind:

• 
  Reporting the result must be displayable to Jenkins. The output that TestExecute/TestComplete now creates is not easy to get into a good way in Jenkins. When an error occurs the details of the error must be displayable in Jenkins preferable with a screenshot.


• 
  Trend reports. Jenkins should able to see the test trend just like unit tests. How many tests/scenarios are there.


• 
  The executable must return proper status code so that Jenkins can interpret the result.

Some things not sure if these should be implemented by the plugin or if this is a TestComplete project setup or other scripting:

• 
  We typically run different kind of tests in different stages in the build. A typical smoke tests which are run on every commit/push then a more thorough test sweet as the build is moved between promotion levels in Jenkins. I think this is something many uses and an Jenkins integration should at least document how this kind of scenario should be handles with TestExecute/TestComplete.


• 
  Initialization of environment. With functional tests there is a need to initialize the environment with test data. This is a problem in any test execution scenario and I'm not sure that a plugin is the right place to handle this, but at least the preferred pratice(s) regarding this problem should be documented.

For me it would be enough this process:

-Jenkins running on any newtwork pc 

-Give Jenkins a command to execute a TC project in ip xxx.xxx.xxx.xxx

-have TC or TE running on that pc



if you have any suggestion please feel free to contribute.

I don't use Jenkins (well, I am not even using TC at work), but I would like to be able to generate report in JUnit-compatible format. It would be useful not only for Jenkins, but also for other CI environments.

"My rant is related to the fact that I tried (and utterly failed) to integrate TC into TeamCity months ago"



Funnily enough, we have a TestComplete project run as part of the nightly build process in TeamCity.



But I suspect my approach is different to yours.



All my logging and reporting is done via script extensions which essentially form a data driven testing framework. This is all driven by user populated excel datasheets. A central script unit acts as the driver for all this. So the script extensions and driver unit form the shell of our automation projects that can easily be used on any new project.



So TeamCity builds the AUT. It then also installs it. Next step is to pull down the repo containing the TestComplete project and run that using TestExecute on a dedicated VM.



The project outputs an HTML file on completion with headline results and a couple of file links. From those file links you can drill down into greater levels of detail for the entire test run. TeamCity interprets both the return code from TestExecute and a couple of elements of the HTML file to indicate overall success or failure.



We find it works really well.



A couple of clever uses of global variable and dummy folders to indicate whether the project is being run in local (development) space or network (build) space as the behaviour varies slightly between the two and it's all portable and easy to use.

Please see my post on LinkedIn for a summary of our results: http://www.linkedin.com/groupItem?view=&gid=1836186&type=member&item=5804626183593152515

Another nice to have:



When TC/TE is run from the command line, it should be able to post messages to stdout and flush the buffer so you can get real-time feedback in Jenkins' console output for a job.