Forum Discussion

wmtan01's avatar
wmtan01
Contributor
5 years ago

Test Execute Summary Report Incomplete

Hi   We're currently using Test Execute to run our smoke tests everyday. We've use the /ExportSummary argument on the CLI for Test Execute and noticed that only the first KWT on a test item collect...
Command Line
Test Results
  • wmtan01's avatar
    wmtan01
    5 years ago

    I got a reply from SmartBear and they have admitted that the issue is still occurring and another fix is scheduled for 14.50 to be released in the next months.

     

    If anyone needs the fix immediately, feel free to reference my Case#'s on SmartBear Customer Care: 00438383 and 00427285 as they are already familiar with the issue if you reference these ones.

     

    As for our case, I won't pursue the fix for now as we have a workaround on our side for this.

wmtan01's avatar
wmtan01
Contributor

We have it configured to stop test item only. Which made the other tests run. we're exporting both MHT and summary and the MHT does have the 5 test items but the summary does not which is what we parse to push results to AzureDevOps for the Pass/Fail flag

TanyaYatskovska's avatar
TanyaYatskovska
SmartBear Alumni (Retired)
5 years ago

thanks for the help, tristaanogre!

 

wmtan01 could you please clarify if the issue still persists? It's unclear from your past reply.

 

  • wmtan01's avatar
    wmtan01
    Contributor
    5 years ago

    Hi,

     

    Sorry for the late response. 

     

    This is how we configured the test items:

    Test Items - We run NAT|EU|LeadManagerProject Configuration

    I believe this should still continue the tests.

    Here's what the summary looks like and what the mht file looks like since we export both:

    MHT File

    Summary XML provided by /ExportSummary

    • wmtan01's avatar
      wmtan01
      Contributor
      5 years ago

      Just an update on this. I'm currently in discussion with Customer Support from Smartbear about this issue.

       

      The Customer Care Engineer was able to replicate the issue and was able to point that it was because we were calling Log.SaveResultsAs in the middle of a collection of test items. We do it after each test run to have faster feedback. He is still in discussion with developers as to why the Log.SaveResultsAs would interfere with the summary generation.

      • palmerbw's avatar
        palmerbw
        Occasional Contributor
        5 years ago

        IS there any update on this? We looked at this same method but noticed the same behavior so we stopped.