TestComplete and Zephyr Enterprise
Hi everyone, This post in mainly for those of you already using both TestComplete and Zephyr Enterprise, or looking into integrating one with the other via the zbot. I'll try to make this discussion have as little gaps in thought as possible, such that I'm not working off of the assumption that everyone is familiar with how the integration works. To begin with, Zephyr Enterprise (ZE) can trigger TestComplete (TC) tests by triggering command line arguments using the zbot (the secure job agent and results parser). The zbot sits on the target location, and when commanded (by ZE) will find the described invocation script path, and use the directed batch file or shell script to launch the automated tests (in our case, the TC tests). It then waits for the execution completion, and looks into another predefined directory for the results file generated by these TC tests in a junit xml format. Attached is what the configuration looks like from the ZE side: Within the dropdown panel that describes the automation framework (where we have selected TestComplete) what this really describes is the junit xml file parser template that the zbot will use to go through our results file to upload our automated TC test results back to ZE. (https://zephyrdocs.atlassian.net/wiki/spaces/ZE/pages/1558445554/Parser+Templates) This means that the zbot will look for certain tags and elements within our junit xml file generated by TestComplete to upload the results back to our test management solution. Typically, a junit xml file generated by TestComplete looks like this: Given the tags, this means that we cannot upload additional information such as attachment files since there is no tags that describe the file location that the supporting docs may reside (i.e the mht files). Now if we wanted to, we could go back in and edit these junit xml files and rerun the tests such that the zbot parses our results (https://zephyrdocs.atlassian.net/wiki/spaces/ZE/pages/1558445158/Vortex+Job+Execution+Events) but that would defeat the purpose of being able to launch our automated TC tests, no? So what we can do is write an additional script, where we have our favorite language of choice run through our junit xml files such that it appends that information for us automatically. This is by no means the right way to do this, just one of the ways! Couple of ways to improve this would be to refactor this script such that it can take additional inputs such as file location (hard coded for now) and also make the attachment subelement recursive so that we can add in multiple attachments. But for now this will do... Now that we have our script that will modify our junit xml results file for us, all we need to do is append a line to our original batch file that triggers our TC job, such that it modifies the generated xml file for us! So the batch file would something like this: where the highlighted portion is just an additional call to the script we wrote above. As you can see the format is familiar for those who have been using command line arguments for TC <TC executable location> <TC pjs. file you want to run> <any additional arugments i.e /r /p /ExportSummary to generate the xml file> <call to our python exe. to run our script> <our python script to append to our xml file> <xml file location ^^> Now our original xml file has been modified to look like this: So now whenever we trigger our TC jobs from within ZE, it will not only carry over the traditional default info such as test case name, run time, failure messages, and execution status, but also the supporting attachment files that we want! More curious users may want to browse our ZE docs to learn how to map it to existing requirements, or add in further information (https://zephyrdocs.atlassian.net/wiki/spaces/ZE/pages/1558445158/Vortex+Job+Execution+Events) On an additional note, we really only went over one portion of the automation integration when it comes to TC and ZE, but we can modify the steps used within this post to apply to the jobs we are running within our CI frameworks as well by adding in an additional build step that will perform the same type of junit xml results file modification we see here. -------------------- How is everyone else utilizing TC with ZE? What additional information do you think could be useful in terms of test management when it comes to your automation jobs? Having supporting attachments, and maybe even automatic code/test coverage is something that can be created with this method, but I'm curious to see what you think could be helpful. Anyhow, I hope everyone is staying safe and sane!
Azure Pipeline - Visual Studio Test - Test execute fails to run (but was working yesterday)
Well, another day another question.😄 Yesterday my build was working fine and my tests were passing on a given project. Today it seems like myVisual StudioTest suddenly fails with the following message: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs /logger:trx "/TestAdapterPath:\"D:\agent\_work\24\s\"" Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. Failed to get a list of tests from the "D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs" file due to the following error: Cannot start process because a file name has not been provided. No test is available in D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. No changes were made to my project and no change to the pipeline itself. It is a bit strange. I tried running testexecute directly on the machine in question and it is running fine. I was wondering if anyone got this issue and could point me in the right direction. I find the logs to be not very verbose, i don't even see how TestExecute is called... 😕
Command line Execution support for Zephyr Folders
When executing test cases in Test Complete or Test Execute via the command line, you can specify which folder to write the test results to but CAN NOT specificy Zephyr Folder. Please add support for Test Complete/Test Execute to write results to Zephyr Folders via command line (or other executions)
Add more log information on Jenkins Smartbear Plugin TestComplete Test Console Output log
Currently there it is just logged with what exe & version & parameters passed. From past experience it would be good to have following details logged too: Configured session details - Run interactive user session flag - Use active session flag - Session screen resolution (sample 2560x1440) The real facts from the Session Desktop during test execution - Is it an interactive user session? - Is an already active session used or did the test runner created a new one? - Session screen resolution (sample 1679x551 which does not match the requested one see above)
Running tests by tag name in specific order
Hello, I will be running my suite of tests in a CI/CD pipeline (Jenkins) but I realized from the documentation that running tests by tag name (following the CLI commands) does not guarantee the tests will be run in a certain order. (Note:The tests that match the specified tag will be run in an arbitrary order.) While trying to find a workaround to this inconvenience, I realized that it does indeed follow a specific order. The order is given by how the test files are sorted in the file script.tcscript. So, for whoever needs to run their tests using tag names, and also wants to run a "pre-step test" or something like that, all you have to do is open that file on any text editor, and move the row where your test file is, up to the place first place, or at whichever place you want it to be executed. Example: # Assuming all tests have the tag "@Smoke" <folder name="MaiGroup"> <folder name="Group1"> <child name="TestMethod1" key="ZZZZ-YYYYY-VVVVV}" path="File1.js" /> <child name="TestMethod2" key="ZZZZ-YYYYY-VVVVV}" path="File2.js" /> <child name="TestMethod3" key="ZZZZ-YYYYY-VVVVV}" path="File3.js" /> </folder> </folder> # Test run for the @Smoke tag will look like: TestMethod1 TestMethod2 TestMethod3 ----------------------------------------------------------- # Now, if we want to run TestMethod3 first, all we have to do is open the script.tcscript file and move the TestMethod3 to the first place: <folder name="MaiGroup"> <folder name="Group1"> <child name="TestMethod3" key="ZZZZ-YYYYY-VVVVV}" path="File3.js" /> <child name="TestMethod1" key="ZZZZ-YYYYY-VVVVV}" path="File1.js" /> <child name="TestMethod2" key="ZZZZ-YYYYY-VVVVV}" path="File2.js" /> </folder> </folder> # And now, the test run for the @Smoke tag will look like: TestMethod3 TestMethod1 TestMethod2 ----------------------------------------------------------- Hope this helps someone, and maybe the TestComplete team can now update the documentation. Regards.
Log Folder Items, Set the Icon while inside the stack (or auto detect on some hiearchy)
I thought this should just be a feature already and I posted asking for help to get it to work. https://community.smartbear.com/t5/TestComplete-General-Discussions/Log-Folder-Icon-Checkpoint-Over-Message-Javascript/m-p/200736#M37434 The log folders are real nice for cleaning up the logs but they only change Icon's if the Folder contains an Error or a Warning. I want LogFolders to show a green if it contains a checkpoint. A common sense order of importance Error > Warning > Checkpoint > Message. Instead it's showing Error > Warning > Message. I think it should even auto detect based on the Log item's priority that has been set. Or just a method to hit ie, Log.CurrentStack(iconWarning, priorityHigh); function TestScriptLog() { Log.AppendFolder("This folder Icon should be an Error"); Log.Message("Informational in Folder","",pmNormal); Log.Checkpoint("Checkpoint in Folder","",pmNormal); Log.Warning("Warning in Folder","",pmNormal); Log.Error("Error in Folder","",pmHighest); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be an Error"); Log.Message("Informational in Folder"); Log.Checkpoint("Checkpoint in Folder"); Log.Warning("Warning in Folder"); Log.Error("Error in Folder"); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be a Warning"); Log.Message("Informational in Folder","",pmNormal); Log.Checkpoint("Checkpoint in Folder","",pmNormal); Log.Warning("Warning in Folder","",pmHighest); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be a Warning"); Log.Message("Informational in Folder"); Log.Checkpoint("Checkpoint in Folder"); Log.Warning("Warning in Folder"); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be a Checkpoint"); Log.Message("Informational in Folder","",pmNormal); Log.Checkpoint("Checkpoint in Folder","",pmHighest); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be a Checkpoint"); Log.Message("Informational in Folder"); Log.Checkpoint("Checkpoint in Folder"); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be an Informational"); Log.Message("Informational in Folder","",pmHighest); Log.PopLogFolder(); Log.AppendFolder("This folder Icon should be an Informational"); Log.Message("Informational in Folder"); Log.PopLogFolder(); }Command line import of TC/TE settings
As someone who is automating the creation of my test environment, I would like to automate the configuration of TestComplete/TestExecute. TestComplete allows for a manual import of a settings file TestExecute does not. This is a problem when I'm attempting to spin up VMs from scratch and immediately start tests. For both products, I would like to be able to use a settings file that I have previously exported to set my desired product settings at the command line either during or after install to finish the product configuration.
Disable the TextComplete\TestExecute update dialoge automatically (without GUI interaction)
Dear Community, I would like to install TestComplete\TestExecute automatically and then call it via the command line to test certain things. The problem I have is the update dialogue. This appears later and interrupts my test. Dialogues can be suppressed with the /SilentMode parameter, but unfortunately, the following call does not work for me with the latest 15.x.x but worked with latest version of 14.x.x with the following parameters. "C:\Program Files (x86)\SmartBear\TestComplete 15\x64\Bin\TestComplete.exe" "C:\TestCompleteProjects\my_solution.pjs" /run /p:Clone_Me /t: "KeywordTests|Main" /SilentMode /ForceConversion /exit Without the "/SilentMode" parameter, everything works perfectly, but the update dialogue appears later. My questions are: - Can I automatically (without disabling it manually in the settings) deactivate the update check without the /SilentMode parameter? - Is it possible to update automatically without the update dialogue appearing? - Why does my call with the /SilentMode parameter not work? Many thanks in advance. Andreas
