How do you use the Log system to track test failures?

This time I am with Marsha :)

I also never met the situation when test log can be taken as it is and used as a final report to the management. We always had to do some manual processing to go through the test logs and to decide if the reported error is an actual problem or something that was caused by the flaw in the test code/system environment/whatever else and that is not reproduced after the failed test is re-executed.

Also, don't forget that real-world automated tests differ from the manual testing. Manual tester is usually aware about latest changes in the tested application and he/she always 'adjusts' the written test to the actual behaviour of the tested application while testing. Usually, he will not report a problem if the current GUI (for example) does not correspond to the one that is mentioned in the manual test if he knows that the GUI was modified in the latest build.

Automated test does not have this knowledge and been a regular deterministic software it will fail and report a problem. Obviously, you can consider two border types of automated tests - very simple and fast created (e.g. recorded ones) and pretty robust that do a lot of work to do what they expected to do and do not fail (e.g. coded tests that use a lot of classes, custom libraries, intelligent search within the tested application, etc.). I think that it is evident, that the former approach will result in a very short time required to implement the test, but in a long run will require a lot of efforts to support such test. Alternatively, the latter approach will result in a significant time required to implement the library code and will require much more time needed to implement the first tests. Tests will be more stable, but the efforts required to support them may depend on the complexity and architecture of the library code. Pretty like when developing a regular application. In a real life we are usually balancing between these border approaches which means that we are always in risk that our test code will fail and require correction.

I don't think that if the automated test fails because of the problem in the test code, the test result should be reported as passed. I think so because the test actually failed. The fact that test failed because of the test code problem may be the reason to report such test as 'blocked and executed manually', but not as passed, because this may create a wrong impression that the given automated test can be executed unattended, while in fact it requires manual execution.



So, for me, the test must be reported in the test log as failed regardless of whether it failed because of the real problem in the tested application of because of some problem with the test code. Then test log must be reviewed and a decision must be made on whether the test failure should be reported with the issue created in the issue tracking system, or the test should be reported as blocked because it requires corrections in the test code.

I see what you are trying to do now.  For us, scripts are just part of the testing, but your process is a bit different.



My first suggestion would be to take some of the time that you are using trying to figure out this logging issue and working around it and fix the actual tests.  I'm sure a lot of us feel like we never have enough time/resources to fix our tests, but I think this would save you time in the long run, even if you just fix the ones that cause you the most trouble.



We have "tests" that actually do setup as well.  They're separate from the testing, so they don't end up in the same log.  If you are running overnight tests, could you run the setup earlier in the day so it doesn't count against your testing?  Or put it in a different project so that TestExecute looks at it separately and you could ignore those logs?



For your script that runs several tests, maybe this will help you pick the log apart to get the items you need:

http://support.smartbear.com/viewarticle/55140/

Hi Scott,



> It reports script failures as test failures, which is incorrect.

Hm-m-m... Why not?

If the script execution failed, what is the reason to report test as succeeded?

I agree, that such failure may be a 'false negative', but this means that in this case script code must be improved to handle the given case. Absolutely like during the regular development of the tested application itself.

I'm with Alexei.  If I get a failure in the log, it's because the application did something we didn't want it to do.



Can you give us an example of a script failure that you wouldn't want to count as an application failure?

The GUI changes.  The script fails because the GUI changed.  Manual testing shows that the GUI does what it is supposed to do.



A database access script changes and introduces a side effect that breaks a script.



The scripts need to be fixed but are not, due to time limitations and staff availability.



There can be issues with QA staff skills.  Some staff are good with the application being tested, but are weak with scripting, so they misinterpret Log error messages and misreport the results.



So far two people have responded that they keep everything in the TC Log system and do not break it out because they do not see a need to.

Maybe a better example would be a setup script.  The setup script does not change test anything. After it finishes running it is reported as a successful test.  This inflates my report of how many tests were performed.  That in turn skews the report of what percentage of the planned tests failed. 



Similarly, some scripts perform multiple tests.  In such a case the log shows that only one test has passed or failed, when really multiple tests have passed or failed.



Scripts are not tests.  TC's log system reports script failures as test failures.

If it's not happening already, I would also suggest that someone familiar with your tests review the programming assignments.  We do this automatically as a part of scrum methodology, but it you are following a different process, try and insert QA early on, so you can see what's coming and then maybe know ahead of time that New Feature A is going to break Test 32 and allow time in the schedule to fix the test.