Forum Discussion

rajs2020's avatar
rajs2020
Frequent Contributor
4 years ago

ReadyAPI - How to repeat failed API tests ?

When *some* API tests fail, then we repeat it (say 3 times) until it succeeds. Currently, we do this by putting a "repeat" script in the tear down script (tab) of every test suite which is prone to f...
  • JKolosova's avatar
    JKolosova
    4 years ago

    Hi rajs2020,

    We currently do not have a feature like that in ReadyAPI, indeed. However, there is the corresponding feature request in our DB, and I have increased its rating according to your message, thanks.

    Using TearDown scripts to rerun failed tests sequentially or in parallel looks to be a good solution for now. For example, you can use a script from the Rerun failed test steps sample project available here:
    https://staging-support.smartbear.com/readyapi/docs/samples/index.html

ChrisAdams's avatar
ChrisAdams
Champion Level 3
4 years ago

"Some tests fail and repeat three times".  I'd argue that either the API is 'buggy' or you have 'flaky' tests.  Either way, personally I would not keep hitting 'run' until I got green bars.  I'd looking at the API and asking the devs why I didn't not get the expected response or I'd be trying to understand why my test fails and what I should do to resolve.

 

You should be looking for consistent behaviour in your tests, then when something does fail, you know you have something that needs to be addressed.

 

The only other scenario i can think of is where the test is failing because of a timing issue.  E.g. Im not getting the expected response as I'm waiting for my earlier data to be persisted, so I'll try again.  For this, I'd suggest a Delay test step.  But again, not ideal and I'd be speaking to the devs if this was the case.

  • HimanshuTayal's avatar
    HimanshuTayal
    Community Hero
    4 years ago

    Hi rajs2020 :

     

    If the tests are failing then there must be a defect there is no logic to run them repeatedly, you should raise a defect.

     

    And if your context is to run your test 3 times i would say you first analyze the failed test cases, if the test cases are failing because of data then you can re-run them else raise the defect.

     

  • rajs2020's avatar
    rajs2020
    Frequent Contributor
    4 years ago

    Yes, repeating tests seems like a problem with the dev code, test code or timing. IMO, repeated failures should be rare and hence tests should only be repeated manually. Automated repeating might mask serious problems. I don't know how such problems could be revealed outside of regression/functional testing. They might get revealed during performance testing or not.

    BUT, I cannot prevent the repetition of failing tests for now. So, I'd like to know how to repeat tests for the short term.

     

    • JKolosova's avatar
      JKolosova
      SmartBear Alumni (Retired)
      4 years ago

      Hi rajs2020,

      We currently do not have a feature like that in ReadyAPI, indeed. However, there is the corresponding feature request in our DB, and I have increased its rating according to your message, thanks.

      Using TearDown scripts to rerun failed tests sequentially or in parallel looks to be a good solution for now. For example, you can use a script from the Rerun failed test steps sample project available here:
      https://staging-support.smartbear.com/readyapi/docs/samples/index.html

      • rajs2020's avatar
        rajs2020
        Frequent Contributor
        4 years ago

        Thanks JKolosova . I'll see that example. Btw, it would be nice to have a cookbook for common ready api use cases. For many programming languages, people have written cookbooks for common tasks like getting the current time in a desired time zone, converting DB result sets to csv files, reading excel sheets etc. Looks like your link on samples is a tiny cookbook of sorts. But, it would be nice to declare it as such, add more content to it and update it regularly.

        BTW, here is an example of a cookbook for the Python language:
        https://www.amazon.com/Python-Cookbook-Third-David-Beazley/dp/1449340377