We use a nightly build and test environment with TestComplete.
The ProjectSuite test run takes 12h and has a huge test result tree.
It is pretty complex to compare the actual ProjectSuite test result with the prevenient test result, to see new divergences.
It would be helpful to see new errors, fixed errors, new tests and not executed tests of a selected test result relating to a other selected test result.
mgroen2 it is still quite coarse. It does not allow you to drill down to individual test items and their reliability over time.
It might be that each test or grouping of tests could be given a GUID so they are uniquely comparable and certainly it would have to be up to the test runner to determine and keep track of these groupings.
There are certainly requirements to define with this feature but it would be invaluable for in helping determine e.g.:
- reliability of the area of the application over time more granular control of defining the metrics for this
- reliability of the tests themselves i.e. do the tests fail as the result of the script logic or other errors e.g. object not found, etc.
- I'm sure there are more
Right now you'd have to create your own scheme and keep track of of this yourself e.g. in a spreadsheet. Very tedious and time consuming.
And once you have a lot of tests the most time consuming part can be going through the logs after the test runs. This would assist in making this process faster.
maximojo I think maybe it's better if TestComplete would provide export function including details of all testitems. To .xlsx or .csv files for example.
So that the comparison (and calculation) can be easily/better done on tools better suited for that (Excel-like programs).
Don't forget TestComplete is a testing tool not a spreadsheet/analysis tool.
I found a solution for us... (just an option for other stakeholders)
Yesterday I've created a TestComplete script method to add a test item name and its result into a csv file.
This method is called by the OnStopTest-Event for each test item and the current result is flagged by the OnLogError-Event.
New test items will be appended to csv file (its result in the correct test run column), disabled test items which are already known in csv file, register a space.
The csv file handling will be executed only when a entire test suite run is in progress (not if only a single test item is executed).
csv file content example:
Testrun ;TR1 ;TR2 ;TR3 ;TR4 ;TR5
TestItem1 ;passed ;passed ;failed ;passed ;passed
TestItem2 ; ;failed ;passed ;passed ;passed
TestItem3 ;passed ; ; ;passed ;failed
TestItem4 ; ; ; ; ;passed
...
At the beginning of a test suite run (or end) TC edits the csv file and adds " ;" to each line = existing but not executed test items are registered - and new results are separated in a new csv column
After a test suite run is finished, I edit the csv file manually with conditional formatting rules, then it looks like the "NetSparker"-screenshot.
To have a Diff Tool for TC is a great idea.
The Result of the diff shall be as hierarchic as the log is.
The diff result shall be accessible for Scripting, so that I can open/Close nodes for the user
