Best approach to testing async API calls?
Good afternoon all,
First post so please be gentle 🙂
Just to give a little context... we're in the process of testing APIs that perform operations asynchronously. Naturally, from a test perspective we need to verify that the asynchronous request has ultimately performed it's desired operation correctly. At present we're just using arbitrary delays awaiting these things to happen but this is far from ideal and adds a level of unreliability to the tests.
1. Rest API call - Giving me a request ID response
2. Delay for X seconds
3. Rest API call - Check status of request ID and should have a JSON attribute indicating success/failure/pending
4. Rest API - Do next operation
What'd like to do is replace step "2 Delay for X seconds" either with... retry step 3 checking the status every Z seconds for a period of Y and do not fail.
Or another approach that meets the needs... we have Soap UI NG Pro in use.
All approaches, suggestions, thoughts welcomed... we haven't dug through all Soap UI NG Pro documentation yet so apologies if it's something obvious in there but haven't seen something that'll meet the need yet.
Appreciate any help! Thanks.
While I would not want to say this is the best way, I do something similar to what you are asking. I do the following:
- Make Initial Call
- Delay Step
- Make Status Check Call (has the async call completed?)
- Groovy TestStep with code to check the output from step 3 and if the async call is not completed/as expected, go back to the delay step and repeat. I also put in a max retry limit to stop things from getting stuck into an infinite loop should something go wrong.
- Validate results are as expected.
You can then tweak the delay time and the amount of retrys to suit you situation.
In real world cases I usually put steps 2,3 & 4 into a separate "common library" test case thus keeping the actual test case a bit cleaner and making the steps to check reusable (as I usually find I'll want to make multiple step 1 "Make Initial Call" with subtly different inputs, that just need slightly different validations in step 5).
That would work perfectly in my context, my Groovy skills are on the novice side at present.
Don't suppose you'd be able to share your script/a sanitised version to give me a step up? Appreciate it.
While I don't have scripts I can share easily at hand, there are a couple of things I can point out with regard to interacting with Ready API to help you on your way.
Firstly have you seen Groovy script Get Data functionality? This will allow you to get data from a previous step using the GUI. This will help you learn the corrrect syntax.
Secondly, you'll see that Groovy test steps provide the script with among others a variable called testRunner this provides several methods of interest to you:
testRunner.gotoStepByName("Test Case Name")
This will allow you to transfer execution to another named test step within the current test case, and
testRunner.fail("Insert your fail reason here.")
This will allow you fail a test for whatever reason you chose, for example if you've reached your self defined retry limit.
Note: If you're just beginning with Groovy scripting, perhaps someone else may be able to offer a better non-Groovy alternative to your question.
Thanks Radford! Appreciate your time... just doing some experimentation at the moment and hopefully will have it sorted reasonably quickly. I've a lot of historical scripting experience so shouldn't be too alien.
Cheers for flagging the get.data and testrunner object... really appreciate your time!
Once I get something simple nailed down I'll prob post a reply here that will hopefully help someone else out down the line.
Can I ask you a quick question?
I've made good progress on a groovy script that meets my need.
Test Script Steps:
Issue POST to API
Delay for Orchestration to complete - 5 seconds
Verify the Status of the automation request
Retry the verification in the event of failure up to timeout - GROOVY SCRIPT
Inside here I'm checking the status of "Verify the Status of the automation request" step
If it failed I'll retry it
If it has passed then perfect!
Has looping logic to retry a certain amount of times etc.
The challenge I'm having is that the TestCase reports failure if I've failed first time through then passed on second iteration. Do I need to override a previous test fail? Can I reset overall testcase status etc? Any guidance you can give here? Or anyone else? 🙂
Appreciate any help
I'm not 100% sure if I follow, but I think I might understand.
When I do my "has the async call completed" check (Step 4 in my post above), if it hasn't completed, I don't actually fail the test, I just use the gotoStepByName("Test Case Name") functionality to jump back to the delay step, which will then rerun the check after the delay, the only additional thing I do is write a log file message to say I've jumped back. I usually set my delay times so on average the retry loop will take about 2 or 3 loop backs before the async call is completed.
I only actually fail the test if my retry loop reaches some abritary high limit that means something serious has gone wrong and it will never complete, usually about 20 retrys.
Hope that make sense?
Note that in my example I am separating out the verifying that the async call has completed (step 4) and has the async call actually done what I want it too (step 5).
That's the issue I'm trying to overcome/presenting me issues.
I.e. my "wait until async call is complete" includes assertions... so I'll either need to figure out a way of working around that or adopt same approach as yours and verify without assertions so we don't trip it into a testcase failure state.
Time to do some more digging... really appreciate your time/thoughts here so far!
As always, there will be lots of ways to solve the problem. Just one more suggestions that may or may not help.
Have you seen the Assertion Test Step? You could remove all of the assertions from your "Verify the Status of the automation request", just checking if the async call is complete in a Groovy test step, then use a assertion test step after the retry loop. I think this might be the most out-of-the-box solution, allowing Ready API to take care of all the checking and reporting on assertions.