Display TestExecute/TestComplete results on TeamCity build log

[sameerjade]

Hello,

I am running my TestComplete tests using the TestExecute command line on TeamCity using batch files. The tests are running ok on TeamCity, however, I do not see any information regarding the passed/failed tests on the TeamCity build log section. On TeamCity it always shows a success irrespective of my tests failing or passing. TestComplete is generating the test results in an html files after running the tests. But I am not sure how to show the test results on the TeamCity build log. I read about the TeamCity service messages here but I am not sure how to use them. Any suggestions? For eg: ##teamcity[testStarted name='testname']

How do I use these messages and where do I enter them?

https://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity#BuildScriptInt...

Thank you.

Regards,

Sameer

[VladYatskovskiy]

Hello. I'm not familiar with TeamCity, but it seems that combining " captureStandardOutput='true' " flag of TeamCity test, and handling TC\TE exit code in your batch files as it described here:
https://support.smartbear.com/testcomplete/docs/working-with/automating/command-line-and-exit-codes/...
may help you.

[garyschechter]

My teammate and I are working on better TeamCity intergration.   Right now what we have built is a custom API and a Python script its a bit run around,  and is still loaded with bugs.  Once we have it more refined we will be releasing it to the community,  I have no ETA on our plans to release it.  

Our product is a web application,  we use a git flow methodolgy and TeamCity builds and deploys to a test enviornment for the branch. 

1. TeamCity's python script tells our custom-API to start tests and the environment.  (http://tcapidev/start?=http://webdev521:52319/ )
2. Our custom API triggers a bat file with "psexec" to launch TestExecute (I would like to switch over to using TE's built in TestRunner API instead of this psexec bat so we have better control to remotely stop the tests).
3. Our TestExecute/TeamCity project has an OnStartTest event,  determins if it's the first test gets the targetURL, then OnLogEvent's to store errors etc,  and then OnStopTest to post the final state and isPassing of recorded logEvents to our custom API.

   (http://tcapidev/post?sessionID=123&testresult=

   {'testname':'welcome page',

   'isPassing': 'true',

   'log':'{'type':'checkpoint',

   'message':'loaded welcomepage'}

   '}  )

4. TeamCity's python script peridocialy checks our in house API for these status messages that TestExecute has posted and gives that to the TamCity build log.

I wish TestComplete's built in API gave this information,  and our Python script could just poll for that instead of having to post it to our own custom API.

[sameerjade]

@VladYatskovskiy Thank you for your reply. Where is the " captureStandardOutput='true' " flag in TeamCity? I could find this flag option in TeamCity UI. 

Thank you.

[sameerjade]

@garyschechter Thank you for your reply. I am hoping that there is an in-built function in TeamCity which can do this and get test results and display it on the build log. Based on this answer, it is possible using the TeamCity service messages feature but its not clear how to use it.

https://stackoverflow.com/questions/9584699/how-to-export-testexecute-testcomplete-results-to-teamci...

[TanyaYatskovska]

Thanks for your suggestions, VladYatskovskiy, garyschechter!

@sameerjade, did you make TestComplete and TeamCity work together?

[sameerjade]

Hello @TanyaYatskovska ,

I am still trying to find a solution for this. I am able to run my TestComplete tests from TeamCity but I cannot get my test results (pass/fail/error) to show on the TeamCity build log page. On TeamCity, the tests always show as passed as long as the script ran ok. Thank you.

Regards,

Sameer

[sameerjade]

I want to add the test run info on the TeamCity build log tab, for eg if my test failed because of a bug and gives an error, I want to show the error on the build log. I tried using TeamCity service messages for this but it is not working as I would expect. Maybe I am missing something or doing it incorrectly. Please advise. All suggestions are welcome. Thank you!

This is how I am using the service messages. I am entering the messages in my .bat file which I am running from TeamCity.

##teamcity[testStarted name='SmokeTest1']
run SmokeTest1.bat
##teamcity[testFailed name='SmokeTest1' message='error']
##teamcity[testFinished name='SmokeTest1']

The above code runs my test and always says that my test failed based on my 'testFailed' command. What I want is that it should check if my test actually failed and report if failed (maybe it can check for some keywords to check if my test failed). But it seems like what is happening is that I am already telling it that the test has failed with the message 'error'. I wouldn't know before running the test if it is going to fail. Am I missing something here?

Also, in the below example, are we supposed to enter the error even before running the test? So, if I use testStdErr  and enter an error, it will always report that error irrespective of if that error really occurred during my test run?

##teamcity[testStarted name='SmokeTest1']

run SmokeTest1.bat

##teamcity[testStdOut name='testname' out='text']

##teamcity[testStdErr name='testname' out='error text']

##teamcity[testFinished name='SmokeTest1']

Thank you!

[TanyaYatskovska · Accepted solution]

Hi @sameerjade,

Have you tried adding the captureStandardOutput parameter to your test as VladYatskovskiy suggested? The following help article contains the info how to do this:

https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html