Forum Discussion

wmtan01's avatar
wmtan01
Contributor
5 years ago

Test Execute Summary Report Incomplete

Hi

 

We're currently using Test Execute to run our smoke tests everyday. We've use the /ExportSummary argument on the CLI for Test Execute and noticed that only the first KWT on a test item collection is being written in the Export Summary:

 

i.e. if we call "NAT|NA"LeadManager" we should be running 5 KWTs but the export summary only says that it ran 1 test, and only the failure of the first of the 5 KWTs is considered for the results.

 

According to the definition of the CLI argument:

"

/ExportSummary:file_name (or /es:file_name) - Commands TestExecute to generate a Summary report for the current test run and to save it in the JUnit report format to the XML file that the file_name parameter specifies.

The report will include the total number of run tests, the number of passed tests and the number of failed tests."

 

Are we missing something or is this actually a bug for Test Execute?

Command Line
Test Results
  • wmtan01's avatar
    wmtan01
    5 years ago

    I got a reply from SmartBear and they have admitted that the issue is still occurring and another fix is scheduled for 14.50 to be released in the next months.

     

    If anyone needs the fix immediately, feel free to reference my Case#'s on SmartBear Customer Care: 00438383 and 00427285 as they are already familiar with the issue if you reference these ones.

     

    As for our case, I won't pursue the fix for now as we have a workaround on our side for this.

  • tristaanogre's avatar
    tristaanogre
    Esteemed Contributor
    5 years ago

    It depends somewhat on what you have your project set to do.  If you have an error written to the log and you have your project and/or test items set to "Stop on Error", than the behavior is expected.  Basically, as soon as the error is logged, the project is stopped so all you get is the results of the test that failed.

    Tools -> Current Project Properties -> Playback has the settings on the project level.

     

    In the project Test Items listing, there are options to indicate what happens for errors on the test item.

    • wmtan01's avatar
      wmtan01
      Contributor
      5 years ago

      We have it configured to stop test item only. Which made the other tests run. we're exporting both MHT and summary and the MHT does have the 5 test items but the summary does not which is what we parse to push results to AzureDevOps for the Pass/Fail flag

      • TanyaYatskovska's avatar
        TanyaYatskovska
        SmartBear Alumni (Retired)
        5 years ago

        thanks for the help, tristaanogre!

         

        wmtan01 could you please clarify if the issue still persists? It's unclear from your past reply.