Test Execute Summary Report Incomplete

[wmtan01]

Hi

We're currently using Test Execute to run our smoke tests everyday. We've use the /ExportSummary argument on the CLI for Test Execute and noticed that only the first KWT on a test item collection is being written in the Export Summary:

i.e. if we call "NAT|NA"LeadManager" we should be running 5 KWTs but the export summary only says that it ran 1 test, and only the failure of the first of the 5 KWTs is considered for the results.

According to the definition of the CLI argument:

"

/ExportSummary:file_name (or /es:file_name) - Commands TestExecute to generate a Summary report for the current test run and to save it in the JUnit report format to the XML file that the file_name parameter specifies.

The report will include the total number of run tests, the number of passed tests and the number of failed tests."

Are we missing something or is this actually a bug for Test Execute?

[tristaanogre]

It depends somewhat on what you have your project set to do.  If you have an error written to the log and you have your project and/or test items set to "Stop on Error", than the behavior is expected.  Basically, as soon as the error is logged, the project is stopped so all you get is the results of the test that failed.

Tools -> Current Project Properties -> Playback has the settings on the project level.

In the project Test Items listing, there are options to indicate what happens for errors on the test item.

[wmtan01]

We have it configured to stop test item only. Which made the other tests run. we're exporting both MHT and summary and the MHT does have the 5 test items but the summary does not which is what we parse to push results to AzureDevOps for the Pass/Fail flag

[TanyaYatskovska]

thanks for the help, tristaanogre!

@wmtan01 could you please clarify if the issue still persists? It's unclear from your past reply.

[wmtan01]

Hi,

Sorry for the late response. 

This is how we configured the test items:

Test Items - We run NAT|EU|LeadManagerProject Configuration

I believe this should still continue the tests.

Here's what the summary looks like and what the mht file looks like since we export both:

MHT File

Summary XML provided by /ExportSummary

[wmtan01]

Just an update on this. I'm currently in discussion with Customer Support from Smartbear about this issue.

The Customer Care Engineer was able to replicate the issue and was able to point that it was because we were calling Log.SaveResultsAs in the middle of a collection of test items. We do it after each test run to have faster feedback. He is still in discussion with developers as to why the Log.SaveResultsAs would interfere with the summary generation.

[palmerbw]

IS there any update on this? We looked at this same method but noticed the same behavior so we stopped. 

[wmtan01]

Hi, we were provided a patch for 14.20. After upgrading to 14.30 we no longer needed that patch and it worked right out of the box on 14.30.

Unfortunately we have not updated to 14.40 yet so I'm not sure if it works there. But 14.30 had no issues.

[wmtan01]

@palmerbw Im sorry to have mislead you, I was referring to a different ticket with the patch. We had an issue before where the contents of .dat files were incomplete and that was the one patched in 14.30.

If you are referring to the summary xml file exported by /ExportSummary, it was supposed to be fixed with 14.40. but I recently tried it and still got an incomplete summary xml. Really sorry about the confusion, I have a bunch of tickets to SmartBear and some of them are affecting the same area like these two that I mixed up. 

I've reopened my SmartBear ticket for the incomplete xml file which is the one I was referring to in this thread. I'll update this thread once they have answered me back.

For reference as well, after discussing it with the SmartBear agent last March, what our company ended up doing was doing 1 test item to run only 1 KWT/Script that way we always got accurate summary xml files which ADOS parses to log results. This also helped us move towards maximizing the parallelization of our tests, but nonetheless this issue should still be fixed by SmartBear.

[wmtan01 · Accepted solution]

I got a reply from SmartBear and they have admitted that the issue is still occurring and another fix is scheduled for 14.50 to be released in the next months.

If anyone needs the fix immediately, feel free to reference my Case#'s on SmartBear Customer Care: 00438383 and 00427285 as they are already familiar with the issue if you reference these ones.

As for our case, I won't pursue the fix for now as we have a workaround on our side for this.