Solved
Test Execute Summary Report Incomplete
Hi We're currently using Test Execute to run our smoke tests everyday. We've use the /ExportSummary argument on the CLI for Test Execute and noticed that only the first KWT on a test item collect...
Forum Discussion
Just an update on this. I'm currently in discussion with Customer Support from Smartbear about this issue.
The Customer Care Engineer was able to replicate the issue and was able to point that it was because we were calling Log.SaveResultsAs in the middle of a collection of test items. We do it after each test run to have faster feedback. He is still in discussion with developers as to why the Log.SaveResultsAs would interfere with the summary generation.
IS there any update on this? We looked at this same method but noticed the same behavior so we stopped.
Hi, we were provided a patch for 14.20. After upgrading to 14.30 we no longer needed that patch and it worked right out of the box on 14.30.
Unfortunately we have not updated to 14.40 yet so I'm not sure if it works there. But 14.30 had no issues.
palmerbw Im sorry to have mislead you, I was referring to a different ticket with the patch. We had an issue before where the contents of .dat files were incomplete and that was the one patched in 14.30.
If you are referring to the summary xml file exported by /ExportSummary, it was supposed to be fixed with 14.40. but I recently tried it and still got an incomplete summary xml. Really sorry about the confusion, I have a bunch of tickets to SmartBear and some of them are affecting the same area like these two that I mixed up.
I've reopened my SmartBear ticket for the incomplete xml file which is the one I was referring to in this thread. I'll update this thread once they have answered me back.
For reference as well, after discussing it with the SmartBear agent last March, what our company ended up doing was doing 1 test item to run only 1 KWT/Script that way we always got accurate summary xml files which ADOS parses to log results. This also helped us move towards maximizing the parallelization of our tests, but nonetheless this issue should still be fixed by SmartBear.
I got a reply from SmartBear and they have admitted that the issue is still occurring and another fix is scheduled for 14.50 to be released in the next months.
If anyone needs the fix immediately, feel free to reference my Case#'s on SmartBear Customer Care: 00438383 and 00427285 as they are already familiar with the issue if you reference these ones.
As for our case, I won't pursue the fix for now as we have a workaround on our side for this.