How to best monitor testing progress

I have struggled with the same problem and I think it is the design of the product that is incorrect not your use of it. 

 

We need to answer the most basic questions for a sprint.  How many tests need to be run for the sprint?  Of those tests, how many have passed, how many have failed and how many still need to be run?  Every day these statistics have to be sent to management.  I was not able to get these statistics from the system as it is. 

 

My solution is this.  At the very beginning of the sprint, before any testing begins, I make very large test sets (for example, 500 tests) of all the tests that we have to run for the sprint.  We have a custom field that identifies the tests needed to be run for a release, so using a filter I can easily identify these tests.  I link each of these test sets to the release iteration and then run each of them.  I immediately end the run as incomplete.  This takes me between 5 and 10 minutes.  What this does is update two fields for each test that are critical for our stats, Last Run Release and Last Run Status (this is set to “Skipped”).  I can then use the dashboard chart called “Tests by Last Run Status”, select the release iteration and I will see the number of tests that I have to run for this iteration shown as “Skipped”.  As people run the tests, the “Skipped” gets replaced by “Failed” or “Passed”.  We have built two additional dashboard charts based on this one that show us tests that have not been run (broken down by assignee) and tests that have failed (broken down by assignee).  The issue with these charts is that if you run the same test for multiple releases or release iterations, then your prior data gets wiped out.  But that is not an issue for us. 

 

It is not a perfect solution, but it works and it gives us what we need.  We have found the vast majority of canned reports to be absolutely useless and we need an easier solution than Crystal Reports.