Forum Discussion

reprosser's avatar
reprosser
Occasional Contributor
4 years ago

Best Practice - Enviroments coverage in Reports

We run all our test cases against OS (windows and linux), http and HTTPS, and Browser (Edge and Chrome). (the test is the same for each option) One way to keep track is to set up environment pick list that contains the permutations - like: Windows-http-Edge, Windows-http-Chrome, Windows-HTTPS-Edge, Windows-HTTPS-Chrome, Linux-http-Edge... etc. Another option is to put custom fields in the test case execution for OS, http, and browser. Which way works best to see what has been covered (or filter) in Reports?

  • MisterB's avatar
    MisterB
    4 years ago

    Hi Rick,

     

    If you were to go with the option of naming the cycles by the environment, you can use the Cycles page itself as a high level dashboard/report - I've attached an example of how I used it in my last project.  There's a lot of useful information in the Cycles page:

     

    - Total tests to execute

    - Issues reported

    - Progress % (with status breakdown when you hover)

    - Cycle status (you can add your own custom statuses such as 'Blocked')

    - And you'll see I included a custom field (Blockage) that I used to report at a high level any issues with the cycle (or environment in your case).

     

    I found it very useful for managing the various test cycles at a glance.

     

    Hope it helps.

     

    Best, Andy

     

  • MisterB's avatar
    MisterB
    Champion Level 3
    4 years ago

    Hi Rick,

     

    You could use a field like Environment, that field actually appears in the standard reports (whereas custom fields do not - and neither do many of the other fields).  A couple of potential downsides to think about:

     

    - Your testers will need to start a new execution for each test case and manually change the environment.  Inconsistencies might lead to confusion in your reporting

    - Tracking multiple environment issues in a single test case could be more complicated to track and monitor (than individual test cases per environment)

    - It won't be apparent to identify how many test executions you really have in total (you have 1 test case being executed 'x' times) - that might make progress and effort reporting more complicated 

     

    Have you considered adding the test cases into separate test cycles that are arranged by environment?  What I mean by that...

    let's say you have 500 test cases.  You can add those test cases into multiple cycles (without duplicating the test cases).  You can then execute, track and report progress by OS.

     

    The potential downside for this approach is that it's less convenient for the testers (if they are thinking of executing 1 case multiple times one after the other in different environments) but if it were me and I was setting things up to make test management and reporting as effective as possible, that's what I'd go for.

     

    Just my opinion of course - and maybe someone else has a better approach (and if so, I'd love to learn from it ğŸ™‚).

     

    Best, Andy

    • reprosser's avatar
      reprosser
      Occasional Contributor
      4 years ago

      Thanks Andy

      We have multiple test cases to run, but looking at just one:

      Currently, we are running a test case with OS/http/Browser settings (from the Enviroment pick list) for each permutation. We pick the environment each time we run the test case. The tester could pick the wrong one, but they have to pick something. So we would run the test case and pick Windows2019-http-Edge as environment. Then we would run the test case and pick Windows2019-http-Chrome as the environment. Then we would pick Windows2019-HTTPS-Edge for the next test run, etc. This allows me to see how many, or when the test case was run against all the envirionment combinations of OS/http/Browser. This requires that all the permutations are listed in the environment pick list - and if another option needs to be covered - like another OS, another Browser, certification type, etc - then the list grows dramatically. This does not seem to be the best approach...

      In this case, I am interested in knowing that the test case has been run against all OS's, both http and HTTPS, and both Edge and Chrome - not in the number of runs, or time estimates. 

       

      One tester would usually run thru all the tests for one environment group (OS/http/browser) and then switch to another run for different environment.

       

      I could create a test cycle, but the cycle does not let us pick an environment, so I would name the cycle to show the OS/http/Browser combo and include the full set of test cases. I don't know how that would work with reports having to track by test cycle names? For this, I would need a test cycle for each permutation (similar to the enviroment pick list). I can have the execution options to fill in for OS/http/Browser on the test case, but that is more work for the tester to keep up with - and may not be needed if the test case name covers it.

       

      Maybe there is a better approach that I have not realized yet. 🙂

       

      • MisterB's avatar
        MisterB
        Champion Level 3
        4 years ago

        Hi Rick,

         

        If you were to go with the option of naming the cycles by the environment, you can use the Cycles page itself as a high level dashboard/report - I've attached an example of how I used it in my last project.  There's a lot of useful information in the Cycles page:

         

        - Total tests to execute

        - Issues reported

        - Progress % (with status breakdown when you hover)

        - Cycle status (you can add your own custom statuses such as 'Blocked')

        - And you'll see I included a custom field (Blockage) that I used to report at a high level any issues with the cycle (or environment in your case).

         

        I found it very useful for managing the various test cycles at a glance.

         

        Hope it helps.

         

        Best, Andy