Forum Discussion

Bob_C's avatar
Bob_C
Contributor
2 years ago

Test Exectution Status for Cases that will not be run

We have a manual test cycle that we run through for a nightly (usually weekly) build that has a total of 25 test cases in it.   We require that our builds go through a complete test cycle, with all test cases passing before it is considered complete. As we run through the cases, if we find a bug, we will typically stop testing at that point, document the bug, and wait for the next build to fix this bug, starting a new cycle with the new build.  If we have an immature software build, we may proceed with the rest of the testing to see if there are other bugs and get this quick feedback to the development team.

If we halt testing, the remaining cases are not technially 'BLOCKED' by the bug, but rather that we decided to not continue testing.  

Is there an industry-standard(ish) term for the status of these test cases that we conciously decide to not test?   'BLOCKED' may the appropriate term, but I would use this more for something like - Step B relies on Step A, and we couldn't complete step A.therefore Step B is blocked, or the hardware failed, and these cases can't be test because of a hardware failure issue.  

 

The end game is that we want to have a report that shows the status of each execution for each build for each test case and visually see how test cases have progressed with each of the builds.

 

Any help is appreciated to get us to this end result.

 

 

 

 

 

 

 

 

 

 

  • MisterB's avatar
    MisterB
    2 years ago

    Interesting question.  Technically, if test case D, E, F, etc. are stopped because test case C has a bug, they are blocked (by your testing process), but I get what you mean and another status could help clarify things further.

     

    I don't know of an industry standard for that scenario but if I were creating a custom one, the ones that spring to mind are Halted or Stopped, but I have also seen and used Skipped, Deferred, Not Applicable, and Descoped.

     

    I think at the end of the day whatever you choose can be reported against and as long as it makes sense to you and everyone else, it'll work for your purposes 🙂

     

    Hope that helps.

  • Bob_C's avatar
    Bob_C
    Contributor
    2 years ago

    Thanks for the reply.  This is what I was looking for.  I had come up with some of those terms already, but thought it is better to ask someone else in the industry what makes the most sense.  

    In this case, I think halted makes the most sense.  There may be some use for skipped in some situations.  I also don't want to get too crazy with these fields, as I have found that the closer you stick to a plain-vanilla system, the better it generally works.

     

     

  • MisterB's avatar
    MisterB
    Champion Level 3
    2 years ago

    Interesting question.  Technically, if test case D, E, F, etc. are stopped because test case C has a bug, they are blocked (by your testing process), but I get what you mean and another status could help clarify things further.

     

    I don't know of an industry standard for that scenario but if I were creating a custom one, the ones that spring to mind are Halted or Stopped, but I have also seen and used Skipped, Deferred, Not Applicable, and Descoped.

     

    I think at the end of the day whatever you choose can be reported against and as long as it makes sense to you and everyone else, it'll work for your purposes 🙂

     

    Hope that helps.

  • MisterB's avatar
    MisterB
    Champion Level 3
    2 years ago

    Glad it helped solidify what you were already thinking.  Something else you might want to consider is creating a custom Test Cycle status for those cycles that are halted - also Halted? - so you can distinguish between test cycles that were completed (Done), and those that were halted. which if left as-is will show as 'In Progress'.

  • Bob_C's avatar
    Bob_C
    Contributor
    2 years ago

    That is a great idea, I think it will help keep things a bit tidier.

    With this, I have a related question - maybe should post separately...  How can I copy/manage these unique Test Execution / Test Cycle Status into all of my different projects?  I have run into this once with Test Cases, where we have added a Status of 'in review' prior to being 'approved'. My workaround was manually typing in all of the information, making sure all details were the same (name, description, status color).  It will not take long before I fat-finger one of these.  In Jira, In creating a new project, I can copy from a base project for workflows etc. but it did not seem to copy over any details for Zephyr.  Maybe I missed a checkbox somewhere?