The responsiveness of a system under test may vary widely depending on the environment. Tests that worked in one run may fail in the next run if the system took longer than expected to respond, then pass again in another run. Such variable responsiveness may be acceptable, depending on product requirements.
Idea:
- Enhance restrequest steps to support retries. Let the test developer specify the max # attempts as well as the delay between attempts (e.g. millisecond).
- Enhance logging to record the #attempts required until test completion (e.g. test step completed after 3 attempts and 3984 milliseconds delay) or similar.
- Set default max #attempts=1, with default delay blank or some value that is grayed out when max #attempts=1. Enable setting of the delay when the user sets max #attempts > 1.
Benefits:
- provide some resilience to functional tests so they continue to succed (from a functional perspective) even when system responsiveness varies
- provide logging information that lets other tools track the evolution of system responsiveness
- allows tests to simulate how real components in a loosely-coupled system would interact with each other (i.e. with retries)
Other thoughts
- The "number of retries" can be confused with the "number of attempts", the latter being one more than the former. So the name for such a capability would have to be chosen and documented carefully.
- Creating a separate step type would avoid modifying the existing restrequest type (less change to existing step).
- However the existing restrequest type could be enhanced in a backwards compatible way and avoids having to explain when to use the old restrequest step and when to use the new one (more transparent).
