cancel
Showing results for 
Search instead for 
Did you mean: 

Test Execute Summary Report Incomplete

SOLVED
Occasional Contributor

Test Execute Summary Report Incomplete

Hi

 

We're currently using Test Execute to run our smoke tests everyday. We've use the /ExportSummary argument on the CLI for Test Execute and noticed that only the first KWT on a test item collection is being written in the Export Summary:

 

image.png

i.e. if we call "NAT|NA"LeadManager" we should be running 5 KWTs but the export summary only says that it ran 1 test, and only the failure of the first of the 5 KWTs is considered for the results.

image.png

 

According to the definition of the CLI argument:

"

/ExportSummary:file_name (or /es:file_name) - Commands TestExecute to generate a Summary report for the current test run and to save it in the JUnit report format to the XML file that the file_name parameter specifies.

The report will include the total number of run tests, the number of passed tests and the number of failed tests."

 

Are we missing something or is this actually a bug for Test Execute?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Community Hero

Re: Test Execute Summary Report Incomplete

It depends somewhat on what you have your project set to do.  If you have an error written to the log and you have your project and/or test items set to "Stop on Error", than the behavior is expected.  Basically, as soon as the error is logged, the project is stopped so all you get is the results of the test that failed.

Tools -> Current Project Properties -> Playback has the settings on the project level.

 

In the project Test Items listing, there are options to indicate what happens for errors on the test item.

2020-02-24 16_04_16-TestComplete - C__MySource_Sandbox_Sandbox.pjs.png


Robert Martin
[Community Expert Group]
Please consider giving a Kudo if I write good stuff
----

Why automate?  I do automated testing because there's only so much a human being can do and remain healthy.  Sleep is a requirement.  So, while people sleep, automation that I create does what I've described above in order to make sure that nothing gets past the final defense of the testing group.
I love good food, good books, good friends, and good fun.

Mysterious Gremlin Master
Vegas Thrill Rider
Extensions available

View solution in original post

5 REPLIES 5
Highlighted
Community Hero

Re: Test Execute Summary Report Incomplete

It depends somewhat on what you have your project set to do.  If you have an error written to the log and you have your project and/or test items set to "Stop on Error", than the behavior is expected.  Basically, as soon as the error is logged, the project is stopped so all you get is the results of the test that failed.

Tools -> Current Project Properties -> Playback has the settings on the project level.

 

In the project Test Items listing, there are options to indicate what happens for errors on the test item.

2020-02-24 16_04_16-TestComplete - C__MySource_Sandbox_Sandbox.pjs.png


Robert Martin
[Community Expert Group]
Please consider giving a Kudo if I write good stuff
----

Why automate?  I do automated testing because there's only so much a human being can do and remain healthy.  Sleep is a requirement.  So, while people sleep, automation that I create does what I've described above in order to make sure that nothing gets past the final defense of the testing group.
I love good food, good books, good friends, and good fun.

Mysterious Gremlin Master
Vegas Thrill Rider
Extensions available

View solution in original post

Highlighted
Occasional Contributor

Re: Test Execute Summary Report Incomplete

We have it configured to stop test item only. Which made the other tests run. we're exporting both MHT and summary and the MHT does have the 5 test items but the summary does not which is what we parse to push results to AzureDevOps for the Pass/Fail flag

Highlighted
Community Manager

Re: Test Execute Summary Report Incomplete

thanks for the help, tristaanogre!

 

@wmtan01 could you please clarify if the issue still persists? It's unclear from your past reply.

 

---------
Tanya Gorbunova
SmartBear Community Manager

 Join the virtual SmartBear Connect user conference on April 27-28! Get your free virtual ticket today!
Highlighted
Occasional Contributor

Re: Test Execute Summary Report Incomplete

Hi,

 

Sorry for the late response. 

 

This is how we configured the test items:

Test Items - We run NAT|EU|LeadManagerTest Items - We run NAT|EU|LeadManagerProject ConfigurationProject Configuration

I believe this should still continue the tests.

Here's what the summary looks like and what the mht file looks like since we export both:

MHT FileMHT File

Summary XML provided by /ExportSummarySummary XML provided by /ExportSummary

Highlighted
Occasional Contributor

Re: Test Execute Summary Report Incomplete

Just an update on this. I'm currently in discussion with Customer Support from Smartbear about this issue.

 

The Customer Care Engineer was able to replicate the issue and was able to point that it was because we were calling Log.SaveResultsAs in the middle of a collection of test items. We do it after each test run to have faster feedback. He is still in discussion with developers as to why the Log.SaveResultsAs would interfere with the summary generation.