Strategy for Disabling Stop on Error
We have made the cardinal sin of creating dependencies between tests. Because of this if test 2 fails the remains 5XX tests aren't run. Because our tests are data driven I believe we can recover from...
Forum Discussion
I adopt the "kill everything and start fresh" approach. For big/fatal errors.
Some of my tests have dependencies on previous tests. In some cases, it's almost unavoidable. It kind of depends what you're doing. In most of these cases they could be made entirely standalone, but the amount of setup work required for each (which the previous tests are at least partly responsible for) would become huge and the runtime would stretch out something silly.
So we accept that sometimes an error sometimes can, and should, cause an abandonment. But without losing the entire run - if possible.
So I deal with this in two ways.
My framework splits tests into "packs". That's a key part of it. Packs should not be huge. And should deal with small, functionally linked, tests.
A "pack" will NEVER have a dependency on a test in another pack. Any dependencies MUST be within the pack. And even the, should be kept to a minimum.
All error handling is done by custom routines. For a simple test failing, I allow tests within a "pack" to have a dependency on previous step(s) within the back. That is to say I can put logic in which allows a test to say "if this passes, do this" / "if this fails do this". This allows me to work round small, known, bugs and the tests will auto-correct themselves when it's fixed but still report the failure while it's not.
But if a larger error occurs, the application crashes being the obvious one, it will kill the application (or website) and all services related to it. Log as much error info as possible, and simply move onto the next "pack". Every pack is designed to start the application/site from a clean baseline and most will have some sort of restore (be it a DB or whatever) or tear-down routine which should always be the first step in any pack. Tests can also be coded to pass back a flag to the main handler which indicate that something terrible has happened and the "pack" should be abandoned.
Seems to work for me. I can work round small errors and defects, and get out of jail with a clean start for bigger ones without losing everything.
Of course, if the application has crashed/broken so badly that it no longer works, even after tear downs and restores, then every subsequent "pack" will simply fail out as well. But there isn't much anyone can do about that (bar re-running installers etc - way too complex) but at least we get a log telling us this. :)
Hi William,
It looks that the approach I used to use is pretty the same as two previous speakers talked about. :)
Basically, every test (or test 'pack' if tests are organized as described by Colin) is responsible to check if the tested application exists and to put it (tested application) into required initial state. If some unexpected windows appear while navigating to the required initial state (usually this happens if something was not saved because of the failed previous test), these windows are attempted to be closed ignoring any changes, just in order not to trap into the same problem that caused the failure of the previous test.
The moment worth to be mentioned here is that the key results of the previous succeeded tests are preserved by my test code and if some test fails the subsequent test tries to use the result of the previous successful test run. For example: if an order was successfully created once, the id of this order is preserved. If during the next test run the order fails to be created, then the next test will use the previous order with the preserved id. (Yes, the orders cannot be created, but why lose the option to be able to check that existing orders still can be viewed and/or printed?)
