Managing Failures - New vs. Expected/Known Issues
Our test suites run on a schedule in multiple environments, using test runner to run tests using tags to run test cases across multiple test suites.
We're struggling to monitor all our test executions, and distinguish between new failures, that may be outages, new bugs, or issue with our tests, and known issues that are bugs that haven't been fixed yet, but we know we've reported and are tracking.
How do you manage distinguishing ReadyAPI failures between ones that need to be looked at and triaged, and ones that are expected failures due to known issues?
I wish there was an additional status for 'warning' in addition to pass/fail.
What other ideas have your teams used to distinguish known issues?
Some ideas, mostly bad -
- tag test cases with known issues (not granular enough as we're seeing issues at a test step level)
- use a script to change a known issue's test step status to 'unknown' instead of fail
Hi mnwill117
Use ReadyAPI’s native fail/pass as the raw result, and handle known issue vs new failure in a separate reporting layer.
- A good approach is a step-level registry keyed by test case, step, environment, and issue ID, so known failures are tagged after execution.
- Avoid changing failed steps to “unknown,” because that hides real failures and weakens trends over time.
Instead, keep execution status unchanged and suppress or route alerts based on whether the failure matches a tracked issue.
For your setup, this is usually the cleanest way to get a warning-like view without altering ReadyAPI’s core result model.
Hope this helps !!
