Forum Discussion

JuanCarlos's avatar
JuanCarlos
New Contributor
27 days ago
Solved

How to Generate a Summary Report for Multiple Test Cycles Without Duplicating Cases

Hello,

We are currently migrating our testing tool from TestLink to Zephyr Scale, but I've encountered a problem with generating our follow-up reports. In TestLink, we used to create reports where we displayed the results of the last version's executions along with the cumulative total of all versions tested.

Here’s the scenario:

I have a Test Plan with 20 test cases, and there are two test cycles (builds) associated with it. In the first cycle, all test cases were executed, with 10 passing and 10 failing. In the second cycle, we only had time to re-execute the 10 failed cases, which now pass.

When generating a summary report in TestLink, I could select both builds, and the graph would display that there are 20 test cases, with all of them passing (10 from the first build and 10 from the second). However, in Zephyr Scale, if I select both test cycles, the report shows 40 test cases (duplicating the test cases), which is not the result I want. I would like Zephyr Scale to behave like TestLink, showing a total of 20 test cases without duplicating the executions.

Is there a way to generate the summary report in Zephyr Scale without summing up the test cases from both test cycles?

Thank you for your help!

Zephyr Scale (general)
  • MisterB's avatar
    MisterB
    26 days ago

    Hi, sorry, I missed a key part of the solution where you need to change reporting to be based on the Test Case Folder location!  This means changing your report to not report by Test Cycle.  This paragraph is now corrected:

    From:
    In my report, using the Advanced Filtering option, in Test Cycles I added the two Cycles, and in Test Execution, chose Iterations v1, and v2, and set the Results output to Last Test Execution (the default value).

    To:
    In my report, using the Advanced Filtering option, Test Cases, I selected the Folder where the tests are stored, and in Test Execution, chose Iterations v1, and v2, and set the Results output to Last Test Execution (the default value).  This solution requires you to file tests logically by folder (which hopefully is what you're already doing), but the advantage is that you can now report executions by folder (i.e. by test case), and show the latest result for those tests regardless of how many test cycles they appear in, e.g. v1, v2, v3, etc.

  • MisterB's avatar
    MisterB
    Icon for Champion Level 3 rankChampion Level 3
    26 days ago

    Part 1

    Hi JuanCarlos, AndreaC,

    This is an interesting scenario.  What you are trying to do is report in the direction of Test Case > Test Execution, whereas I think Scale is more suited to reporting based on Test Execution as the starting point.  But I believe it's still possible to report in the way you want 😊

    In my tests, after replicating your scenario, I went to the Configuration menu and added two Iterations, e.g. v1, v2.  I then assigned v1 to all tests in Cycle One (10 pass, 10 fail), and assigned v2 to 10 tests that passed in Cycle 2 (leaving the 10 not executed without an iteration).

    In my report, using the Advanced Filtering option, in Test Cycles I added the two Cycles, and in Test Execution, chose Iterations v1, and v2, and set the Results output to Last Test Execution (the default value).

    This worked for me - I got 20 results in total, showing the latest result for all executions in v1 & v2.  You may still encounter issues because the report is based on all executions in those cycles, so another option is to swap Test Cycles filter criteria and instead use Test Case > Folder criteria (making sure that your test cases are structured in folders how you would like them reported).

    I think there is another way to do this - possibly cleaner - and will report back on that in Part 2 😊.

    Let me know if this works for you?

    • JuanCarlos's avatar
      JuanCarlos
      New Contributor
      26 days ago

      Thank you very much for your help. After reviewing both cases, the first one would be the best fit for our way of working. However, after following your suggestion (assigning iteration 1 to all test cases from the first test cycle, whether they passed or failed, and iteration 2 only to those executed in the second test cycle), when applying the filter as per your instructions, it displays all the test cases executed in iteration 1 plus those executed in iteration 2.

      • MisterB's avatar
        MisterB
        Icon for Champion Level 3 rankChampion Level 3
        26 days ago

        Hi, sorry, I missed a key part of the solution where you need to change reporting to be based on the Test Case Folder location!  This means changing your report to not report by Test Cycle.  This paragraph is now corrected:

        From:
        In my report, using the Advanced Filtering option, in Test Cycles I added the two Cycles, and in Test Execution, chose Iterations v1, and v2, and set the Results output to Last Test Execution (the default value).

        To:
        In my report, using the Advanced Filtering option, Test Cases, I selected the Folder where the tests are stored, and in Test Execution, chose Iterations v1, and v2, and set the Results output to Last Test Execution (the default value).  This solution requires you to file tests logically by folder (which hopefully is what you're already doing), but the advantage is that you can now report executions by folder (i.e. by test case), and show the latest result for those tests regardless of how many test cycles they appear in, e.g. v1, v2, v3, etc.

    • JuanCarlos's avatar
      JuanCarlos
      New Contributor
      26 days ago

      If there is no filter available for this, is there any way to group by test case? The idea would be to group test cases with the same name and only display the latest execution for each of them.

      • MisterB's avatar
        MisterB
        Icon for Champion Level 3 rankChampion Level 3
        26 days ago

        Yes, that's exactly the approach you're looking for and which I have provided in my revised answer

  • MisterB's avatar
    MisterB
    Icon for Champion Level 3 rankChampion Level 3
    26 days ago

    Part 2

    Another way to achieve the same result without using two Test Cycles is to use the "Start a new test execution" method (see image 1 below).  This enables you to have one Test Cycle and execute a test multiple times (e.g. as versions are released), updating your reports and charts.  If reporting by version is important, you could use either Versions or Iterations for that purpose and be able to show test results and charts by version and overall status.

    How it would work:

    1. Create one Test Cycle with your 20 tests
    2. Execute those tests recording the Version of Iteration, e.g. v1
    3. You have 10 Pass, and 10 Fail all against v1 and are able to report the status of v1 testing
    4. Now we move onto v2, and are ready to re-test the 10 Failed tests
    5. For each Failed test we click the "Start a new test..." button and record that we're in v2 (Iteration or Version), and set the status to Pass
    6. We now have 20 tests with a Pass result, and can also report that we only executed 10 tests in v2
    7. And we can also run a report for the Test Cycle showing that we executed either 20 tests (all Pass), or 30 tests (20 Pass, 10 Fail).

    Image 1:

     

     

  • AndreaC's avatar
    AndreaC
    New Contributor
    27 days ago

    Hope this gets a reply... my team has also been trying to figure out how to generate a report without adding up the same tests but to show a different overall results... Thanks for posting!