Forum Discussion

rajs2020's avatar
rajs2020
Frequent Contributor
4 years ago

ReadyAPI - How to repeat failed API tests ?

When *some* API tests fail, then we repeat it (say 3 times) until it succeeds. Currently, we do this by putting a "repeat" script in the tear down script (tab) of every test suite which is prone to failures. It is cumbersome to repeat the script for every problematic suite & update the script.

1 - Is it a good idea to repeat API tests until they pass?
2 - Are there better ways of repeating API tests ?
3 - Why does ready api (version 3.0) not have the option to easily repeat API tests ?

  • JKolosova's avatar
    JKolosova
    4 years ago

    Hi rajs2020,

    We currently do not have a feature like that in ReadyAPI, indeed. However, there is the corresponding feature request in our DB, and I have increased its rating according to your message, thanks.

    Using TearDown scripts to rerun failed tests sequentially or in parallel looks to be a good solution for now. For example, you can use a script from the Rerun failed test steps sample project available here:
    https://staging-support.smartbear.com/readyapi/docs/samples/index.html

  • ChrisAdams's avatar
    ChrisAdams
    Champion Level 3
    4 years ago

    "Some tests fail and repeat three times".  I'd argue that either the API is 'buggy' or you have 'flaky' tests.  Either way, personally I would not keep hitting 'run' until I got green bars.  I'd looking at the API and asking the devs why I didn't not get the expected response or I'd be trying to understand why my test fails and what I should do to resolve.

     

    You should be looking for consistent behaviour in your tests, then when something does fail, you know you have something that needs to be addressed.

     

    The only other scenario i can think of is where the test is failing because of a timing issue.  E.g. Im not getting the expected response as I'm waiting for my earlier data to be persisted, so I'll try again.  For this, I'd suggest a Delay test step.  But again, not ideal and I'd be speaking to the devs if this was the case.

    • HimanshuTayal's avatar
      HimanshuTayal
      Community Hero
      4 years ago

      Hi rajs2020 :

       

      If the tests are failing then there must be a defect there is no logic to run them repeatedly, you should raise a defect.

       

      And if your context is to run your test 3 times i would say you first analyze the failed test cases, if the test cases are failing because of data then you can re-run them else raise the defect.

       

    • rajs2020's avatar
      rajs2020
      Frequent Contributor
      4 years ago

      Yes, repeating tests seems like a problem with the dev code, test code or timing. IMO, repeated failures should be rare and hence tests should only be repeated manually. Automated repeating might mask serious problems. I don't know how such problems could be revealed outside of regression/functional testing. They might get revealed during performance testing or not.

      BUT, I cannot prevent the repetition of failing tests for now. So, I'd like to know how to repeat tests for the short term.

       

      • JKolosova's avatar
        JKolosova
        SmartBear Alumni (Retired)
        4 years ago

        Hi rajs2020,

        We currently do not have a feature like that in ReadyAPI, indeed. However, there is the corresponding feature request in our DB, and I have increased its rating according to your message, thanks.

        Using TearDown scripts to rerun failed tests sequentially or in parallel looks to be a good solution for now. For example, you can use a script from the Rerun failed test steps sample project available here:
        https://staging-support.smartbear.com/readyapi/docs/samples/index.html