Ask a Question

Retry based on AssertionStep status

SOLVED
pb1
Contributor

Retry based on AssertionStep status

Hi there

 

Intro here, the actual problem is below the line. We are currently working on a retry-mechanism that re-runs a specific interval of testSteps upon failure. 

 

pb1_0-1597653921353.png

In the given picture the test would transfer a file to a server, which extends an already existing list with new data. Because the processing system is outside of our scope we can only wait and guess when the server is done ingesting and processing the data. In the current project we wait for a static 2 Minutes (and it still takes longer sometimes).

The entire logic would be to simulate a testRunner and have the script run all steps from index 2 to 5, check the assertion step at index 5 and then continue the testCase. Upon failure it would run all steps from index 3 to 5 again.

 

The pure logic is rather basic and would look like this:

 

 

void ftpRetry(def testRunner, def context, int retries){
  List retryCheckpoints = ['retryCheckpoint']
  int currentIndex = context.getCurrentStepIndex()
  ( checkpointStep, checkpointIndex ) = findStepInTestCase(context, false, retryCheckpoints)
  Boolean passCheckpoint = false
  while(retries-- > 0 && !passCheckpoint){
    runTestSteps(testRunner, context, currentIndex + 1, checkpointIndex)
    if(checkpointStep.getAssertionStatus().toString() == 'PASS') {
      passCheckpoint = true
    }
  }
  assert passCheckpoint : "Out of retries, checkpoint-assertion failed every time!"
}

void runTestSteps(def testRunner, def context, int startIndex, int endIndex) {
  while(startIndex <= endIndex) {
    context.getTestCase().getTestStepAt(startIndex++).run(testRunner, context)
  }
}

 

 

________________________________________________________________________________________

 

The script looks fine to me, but calling .getAssertionStatus() on the AssertionStep always returns "PASS". It appears as if the step itself runs, but the assertions within the step are not running because their status is always "UNKNOWN". This may have something to do with the AssertionStep itself, but we currently can not use internal assertions in REST-Steps themselves. We must put assertions in their own step!

I am currently crawling through the API to figure out why the assertions within the assertionStep don't want to run, but I'd like to see if someone can provide a solution.

1 ACCEPTED SOLUTION

Accepted Solutions
pb1
Contributor

Re: Retry based on AssertionStep status

I managed to solve this specific problem, the opened support request shall remain open because I still suspect a problem with the .run() function.

 

Instead of trying to emulate the testRunner from inside the groovy step we now "drop an anchor" before the retry-cycle, run into the script and check the checkpoint status. If the status is FAIL we tell the testRunner to go back to the anchor via testRunner.gotoStep() - if the status is PASS we just let the TestCase continue on. Here is how that looks:

pb1_0-1598611173343.png

The anchor contains the number of remaining retry attempts, which are subtracted with each iteration by the ftpRetry-Script.

pb1_1-1598611237079.png

And here is the adapted ftpRetry function from the example.

 

void ftpRetry(def testRunner, def context){
  // variables to allow user to specify other names
  List<String> retryCheckpointNames = ['retryCheckpoint']
  List<String> retryAnchorNames = ['retryAnchor']
  int defaultRetryCount = 15
  ( checkpointStep, checkpointIndex ) = findStepInTestCase(context, true, retryCheckpointNames)
  ( anchorStep, anchorIndex ) = findStepInTestCase(context, true, retryAnchorNames)
  if(checkpointStep.getAssertionStatus().toString() != 'PASS') {
    // checkpoint did not pass, get the anchor, set the testRunner to the index of the anchor and subtract a retry from the anchor, then let everything run until this script is called again
    int retries = anchorStep.getPropertyValue('retries').toInteger()
    if(retries-- > 0) {
      anchorStep.setPropertyValue('retries', retries.toString())
      testRunner.gotoStep(anchorIndex + 1)
    }
    else {
      anchorStep.setPropertyValue('retries', defaultRetryCount.toString())
      assert false : "Out of retries, checkpoint-assertion failed every time!"
    }
  }
  else {
    log.info("Checkpoint passed, resetting retries on anchor.")
    anchorStep.setPropertyValue('retries', defaultRetryCount.toString())
  }
}

 

 

Should have reconsidered my approach to the problem rather than trying to force the script to emulate the testRunner, oh well.

View solution in original post

7 REPLIES 7
HimanshuTayal
Community Hero

Re: Retry based on AssertionStep status

@pb1 : Try with .getStatus() method instead of getAssertionStatus().

 

Hope it will Help 🙂

 

 

 


Click "Accept as Solution" if my answer has helped,
Remember to give "Kudos" 🙂 ↓↓↓↓↓



Thanks and Regards,
Himanshu Tayal
pb1
Contributor

Re: Retry based on AssertionStep status

@HimanshuTayal Same problem sadly.

pb1_0-1597655890953.png

I can guarantee that the step called has assertions can would lead to a failure, I am currently assuming that calling .run() just doesn't actually execute the assertions, so the "empty" AssertionStep is being executed and 0 assertions = 0 fails, so the step itself passes.

HimanshuTayal
Community Hero

Re: Retry based on AssertionStep status

@pb1 : Write like below

 

runner = step.run(testRunner,context);
log.info runner.getStatus()

 

Hope this will work 🙂


Click "Accept as Solution" if my answer has helped,
Remember to give "Kudos" 🙂 ↓↓↓↓↓



Thanks and Regards,
Himanshu Tayal
pb1
Contributor

Re: Retry based on AssertionStep status

@HimanshuTayal 

Thanks for the help so far, but sadly your code produces a "PASS" as well.

pb1
Contributor

Re: Retry based on AssertionStep status

At this point I am 100% sure that calling .run() on the AssertionTestStep does not execute any AssertionEntry. 

I can not figure out how to emulate the actual testRunner, my best attempt to run the entries follows: 

def assertionStep = context.getTestCase().getTestStepAt(context.getCurrentStepIndex() + 4)
assertionStep.getAssertionEntryList()
	.each{ assertionEntry ->
		assertionEntry.getAssertion().assertRequest(?????, context)
	}.collect { assertionEntry ->
		assertionEntry.getStatus()
	}

The problem right now is that assertRequest() requires the MessageExchange object, but I have no idea how to get the MessageExchange from the previous RestRequest, the API is very undocumented and after a day of digging I could not find a way to make it work.

sonya_m
Community Manager

Re: Retry based on AssertionStep status

@pb1 I see that you are working with Support on this (case 00448152).

Please keep the investigation going. Also, we would really appreciate it if you post your solution here that you get from Support. Thank you.


Sonya Mihaljova
Community and Education Specialist

Did my reply answer your question? Give Kudos or Accept it as a Solution to help others. ⬇️⬇️⬇️
pb1
Contributor

Re: Retry based on AssertionStep status

I managed to solve this specific problem, the opened support request shall remain open because I still suspect a problem with the .run() function.

 

Instead of trying to emulate the testRunner from inside the groovy step we now "drop an anchor" before the retry-cycle, run into the script and check the checkpoint status. If the status is FAIL we tell the testRunner to go back to the anchor via testRunner.gotoStep() - if the status is PASS we just let the TestCase continue on. Here is how that looks:

pb1_0-1598611173343.png

The anchor contains the number of remaining retry attempts, which are subtracted with each iteration by the ftpRetry-Script.

pb1_1-1598611237079.png

And here is the adapted ftpRetry function from the example.

 

void ftpRetry(def testRunner, def context){
  // variables to allow user to specify other names
  List<String> retryCheckpointNames = ['retryCheckpoint']
  List<String> retryAnchorNames = ['retryAnchor']
  int defaultRetryCount = 15
  ( checkpointStep, checkpointIndex ) = findStepInTestCase(context, true, retryCheckpointNames)
  ( anchorStep, anchorIndex ) = findStepInTestCase(context, true, retryAnchorNames)
  if(checkpointStep.getAssertionStatus().toString() != 'PASS') {
    // checkpoint did not pass, get the anchor, set the testRunner to the index of the anchor and subtract a retry from the anchor, then let everything run until this script is called again
    int retries = anchorStep.getPropertyValue('retries').toInteger()
    if(retries-- > 0) {
      anchorStep.setPropertyValue('retries', retries.toString())
      testRunner.gotoStep(anchorIndex + 1)
    }
    else {
      anchorStep.setPropertyValue('retries', defaultRetryCount.toString())
      assert false : "Out of retries, checkpoint-assertion failed every time!"
    }
  }
  else {
    log.info("Checkpoint passed, resetting retries on anchor.")
    anchorStep.setPropertyValue('retries', defaultRetryCount.toString())
  }
}

 

 

Should have reconsidered my approach to the problem rather than trying to force the script to emulate the testRunner, oh well.

View solution in original post

cancel
Showing results for 
Search instead for 
Did you mean: 

Join the Community

Ask Community Experts

Join more than 100,000 users worldwide. Ask questions, request features, post replies, award kudos, and more.

Join Now For Free

Already a member? Sign In