ReadyAPI - How to repeat failed API tests ?

[rajs2020]

When *some* API tests fail, then we repeat it (say 3 times) until it succeeds. Currently, we do this by putting a "repeat" script in the tear down script (tab) of every test suite which is prone to failures. It is cumbersome to repeat the script for every problematic suite & update the script.

1 - Is it a good idea to repeat API tests until they pass?
2 - Are there better ways of repeating API tests ?
3 - Why does ready api (version 3.0) not have the option to easily repeat API tests ?

[ChrisAdams]

"Some tests fail and repeat three times".  I'd argue that either the API is 'buggy' or you have 'flaky' tests.  Either way, personally I would not keep hitting 'run' until I got green bars.  I'd looking at the API and asking the devs why I didn't not get the expected response or I'd be trying to understand why my test fails and what I should do to resolve.

You should be looking for consistent behaviour in your tests, then when something does fail, you know you have something that needs to be addressed.

The only other scenario i can think of is where the test is failing because of a timing issue.  E.g. Im not getting the expected response as I'm waiting for my earlier data to be persisted, so I'll try again.  For this, I'd suggest a Delay test step.  But again, not ideal and I'd be speaking to the devs if this was the case.

[HimanshuTayal]

Hi @rajs2020 :

If the tests are failing then there must be a defect there is no logic to run them repeatedly, you should raise a defect.

And if your context is to run your test 3 times i would say you first analyze the failed test cases, if the test cases are failing because of data then you can re-run them else raise the defect.

[rajs2020]

Yes, repeating tests seems like a problem with the dev code, test code or timing. IMO, repeated failures should be rare and hence tests should only be repeated manually. Automated repeating might mask serious problems. I don't know how such problems could be revealed outside of regression/functional testing. They might get revealed during performance testing or not.

BUT, I cannot prevent the repetition of failing tests for now. So, I'd like to know how to repeat tests for the short term.

 

[JKolosova · Accepted solution]

Hi @rajs2020,

We currently do not have a feature like that in ReadyAPI, indeed. However, there is the corresponding feature request in our DB, and I have increased its rating according to your message, thanks.

Using TearDown scripts to rerun failed tests sequentially or in parallel looks to be a good solution for now. For example, you can use a script from the Rerun failed test steps sample project available here:
https://staging-support.smartbear.com/readyapi/docs/samples/index.html

[rajs2020]

Thanks @JKolosova . I'll see that example. Btw, it would be nice to have a cookbook for common ready api use cases. For many programming languages, people have written cookbooks for common tasks like getting the current time in a desired time zone, converting DB result sets to csv files, reading excel sheets etc. Looks like your link on samples is a tiny cookbook of sorts. But, it would be nice to declare it as such, add more content to it and update it regularly.

BTW, here is an example of a cookbook for the Python language:
https://www.amazon.com/Python-Cookbook-Third-David-Beazley/dp/1449340377