Forum Discussion

towardsQuality's avatar
towardsQuality
Occasional Contributor
11 months ago

How to change Test Case Status using Groovy

Hi!

There is some integration and timing uncertainty within the product under test that has required me to utilize some Groovy code to retry a test step (a REST GET call) x number of times, with y amount of time between each try. 

As soon as RAPI encounters a test step failure, it sets the entire test case to FAIL. But in my case, subsequent runs of the above noted test step may pass. This leaves a test case with all passing test cases, but a failed test.

I want to be able to set the test case to PASS if this scenario occurs. I've tried multiple things and done a bunch of web searches, but am stuck. Any help would be MUCH appreciated!

Here the code I'm using in my teardown step:

def testCase = testRunner.testCase
def testSteps = testCase.getTestStepList()
def testStatus = "PASS"
 
for( runner in testSteps ) {
    if ( runner instanceof com.eviware.soapui.impl.wsdl.teststeps.RestTestRequestStep ) {
    if ( runner.getAssertionStatus().toString() == "FAIL" ) {
    testStatus = "FAIL"
    break
    }
     }
}
 
if ( testStatus == "PASS" ) {
  <<set test CASE status to PASS>>
}
 
NOTES:
1) is there a good resource that would help me better understand all the RAPI functions available using Groovy in RAPI? My scripting language of choice (and experience) is python. I'm just now learning the Groovy/Java world.
2) Things I've tried that haven't worked (thrown a java error):
a)
import com.eviware.soapui.model.testsuite.TestRunner.Status
testCase.setStatus(Status.FINISHED)
 
b)
testRunner.testCase.setStatus(com.eviware.soapui.model.testsuite.TestRunContext.ResultStatus.PASSED)
 
c)
testCase.setPropertyValue("Status", "PASSED")
 
d) 
import com.eviware.soapui.model.testsuite.TestRunContext.ResultStatus
testCase.setStatus(ResultStatus.PASSED)
Scripting
  • nmrao's avatar
    nmrao
    11 months ago

    towardsQuality 

    After looking at all the responses,

    Current steps:

    POST REST Request

    Groovy Script with retry logic

    GET REST Request

    Here is what I would  suggest or do in this scenario.

    Changes of sequence:

    POST REST Request

    GET REST Request (Disable the step)

    Groovy script with retry logic

    Reason being that the execution control will be at Groovy Script step and still be able to run the GET call. Also avoid stopping the test execution by disabling the GET call step.

    And there is no need to have any logic for Tear Down script of the test case as well.

    Assuming that the test steps are in same sequence and there is no other steps in between these.

    And here is the groovy script content goes:

    import com.eviware.soapui.model.testsuite.Assertable.AssertionStatus
import com.eviware.soapui.model.testsuite.TestStepResult.TestStepStatus
def maxAttempts = 7
def waiting = 10
def doubleIt = { it * 2 }
def currentWait = 1

//Get the test step just before the current groovy script step, which is GET REST call step
def step = context.testCase.testStepList[context.currentStepIndex-1]
def isStepFailed = {
	log.info "Executing test step : ${step.name}"
	def result = step.run(testRunner, context)
	log.info "Assertion : ${step.assertionStatus}"
	log.info "Step result: ${result.status}"
	result.status == TestStepStatus.FAILED ? true : false
}

/**
* This will first wait for 10 sec, then 20 sec, then 40 sec so that number calls will be reduced, you can change it as needed
*/
def waitFor = { 
     currentWait = (it == 1) ? currentWait : doubleIt(currentWait)
     log.info "Waiting for ${waiting * currentWait} seconds before attempting ${step.name} again"
    Thread.sleep(1000 * waiting * currentWait)   
}

for(item in 1..maxAttempts) {
	waitFor(item)
	log.info "Attempt: $item"
	if (!isStepFailed()) {
		break
	} else {
               //Test will be failed once GET call fails after exhaustings maxAttempts
		if (item == maxAttempts) assert false, "GET instance failed after $item attempts"
	}
}
return

    Please give it a try and see if it helps!

    By the way, no need to change the Test Case options unlike discussed earlier, keep the default

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    Before going for Groovy way, check if below helps!

    There are Test Case options available, please see the documentation 

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      Thanks for your reply, nmrao!

      The only option that seems to apply in my scenario is the 'Fail Test Case on Error' option. I'm concerned about the unintended consequences of this option. In general, we do want tests to fail if a test step fails. But there are some corner cases where we don't.

      Thus, I'd prefer a Groovy solution. Isn't there a simple way to set the test case status to 'PASS' using Groovy?

      NOTE: I tried deselecting the 'Fail Test Case on Error' option and it doesn't seem to have an effect: my test case continues to fail as soon as a test step fails, even though I have Groovy retry logic that reruns the test step and it eventually passes.

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    That's the option I want you to look at it.

    The only difference when unchecked is all the steps are run instead stopping at the failed step.

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      Thank you, nmrao.

      That option doesn't solve my problem. I need to be able to:

      • run a test step
      • if fail, wait a number of secs (defined in a Groovy script)
      • rerun test step
      • if pass after predefined number of retries (configured in Groovy script), then set test case status back to PASS (assuming no other test steps failed)

      The best way I know to do this is to have a test CASE teardown step that looks at the final status of each test STEP, and if they are all PASS, then set the test CASE to PASS as well. But I can't figure out how to set the test case to PASS using Groovy.

      Thanks for any advice!

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    Regarding retrying the failed tests, another  approach can be to execute only failed tests again later instead of repeating the steps in the same run.

    Of course, here need to come up with an automated way to run them as well using scripting.

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      Appreciate this idea. In my case, our systems are integrated with multiple other components. There are often timing issues that pass the timing acceptance criteria, but may take between 10 seconds and 90 seconds to return results. The setup of the tests is quite complex and the test STEPs can't be run independently. In some instances, I can add a 90 second WAIT. But that is wasting test run execution time when the result is available in 10 seconds. In other circumstances, I need to know within 10 seconds when the intended results are returned so that I can run the next step of the test case.

      Thus, I'm very happy with my retry logic... I'm just NOT happy that I can't reset the test CASE status to pass when subsequent attempts succeed.

      • nmrao's avatar
        nmrao
        Champion Level 3
        11 months ago

        I didn't mean to repeat just the step, instead whole test case(s) (failed).

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    What is happening when you try to check the status of each step in groovy? 

    Is the test case not automatically failing? Why you wanted to update using groovy?

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      I am able to check the status of each step in Groovy successfully. 

      The test case is automatically failing as soon as a test step fails. In most cases, that makes sense. But in about 10% of the cases I'm working with, I need to RETRY the test step several times before is succeeds. 

      The problem is that the FIRST time the test step fails, it marks the test case as FAIL. And I can't figure out how to mark the test case back to PASS, assuming that subsequent runs of the test step eventually pass.

      From my first post, this is the code I'm looking for:

      <<set test CASE status to PASS>>

      • nmrao's avatar
        nmrao
        Champion Level 3
        11 months ago

        What is the typical sequence of steps involved in your 10% tests which you are trying now?

        What if you repeat the whole test case instead of just the test step if possible (as it is not allowing to change the status of the test case after it failed)?

        How are you currently repeating the failed test step? 

        If possible, screen shot of test case, interested in the test step types and their sequence.

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    towardsQuality 

    After looking at all the responses,

    Current steps:

    POST REST Request

    Groovy Script with retry logic

    GET REST Request

    Here is what I would  suggest or do in this scenario.

    Changes of sequence:

    POST REST Request

    GET REST Request (Disable the step)

    Groovy script with retry logic

    Reason being that the execution control will be at Groovy Script step and still be able to run the GET call. Also avoid stopping the test execution by disabling the GET call step.

    And there is no need to have any logic for Tear Down script of the test case as well.

    Assuming that the test steps are in same sequence and there is no other steps in between these.

    And here is the groovy script content goes:

    import com.eviware.soapui.model.testsuite.Assertable.AssertionStatus
import com.eviware.soapui.model.testsuite.TestStepResult.TestStepStatus
def maxAttempts = 7
def waiting = 10
def doubleIt = { it * 2 }
def currentWait = 1

//Get the test step just before the current groovy script step, which is GET REST call step
def step = context.testCase.testStepList[context.currentStepIndex-1]
def isStepFailed = {
	log.info "Executing test step : ${step.name}"
	def result = step.run(testRunner, context)
	log.info "Assertion : ${step.assertionStatus}"
	log.info "Step result: ${result.status}"
	result.status == TestStepStatus.FAILED ? true : false
}

/**
* This will first wait for 10 sec, then 20 sec, then 40 sec so that number calls will be reduced, you can change it as needed
*/
def waitFor = { 
     currentWait = (it == 1) ? currentWait : doubleIt(currentWait)
     log.info "Waiting for ${waiting * currentWait} seconds before attempting ${step.name} again"
    Thread.sleep(1000 * waiting * currentWait)   
}

for(item in 1..maxAttempts) {
	waitFor(item)
	log.info "Attempt: $item"
	if (!isStepFailed()) {
		break
	} else {
               //Test will be failed once GET call fails after exhaustings maxAttempts
		if (item == maxAttempts) assert false, "GET instance failed after $item attempts"
	}
}
return

    Please give it a try and see if it helps!

    By the way, no need to change the Test Case options unlike discussed earlier, keep the default

    • nmrao's avatar
      nmrao
      Champion Level 3
      11 months ago

      Currently, uses def waiting = 10 which is multiples of 10 seconds.

      If you want to reduce the waiting time, try reducing value say  to 5, so that it will wait start with 5 sec delay, then 10, then 20 etc.

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      Hmm... This solution is looking promising.

      I am getting a java error on the 'assert false':

      java.lang.AssertionError: GET instance failed after 2 attempts. Expression: false
      error at line: 34

      Any idea how to fix above?

      And for future reference, are you aware of a way to change the test cases status (from pass to fail or vice-versa) using Groovy?

      • nmrao's avatar
        nmrao
        Champion Level 3
        11 months ago

        You can also look at the script log for more details.

  • nmrao's avatar
    nmrao
    Champion Level 3
    11 months ago

    Have you tried the script as Is? or made any changes?

    Hope that you can see and understand what it is doing based on the comments.

    It is clear from the error message that "GET Rest call" is failed after 2 attempts which you might set the maxAttempts. I believe, this is what you mentioned in the previous responses i.e., fail the test case after retrying the GET REST call for the given number of attempts.

    Script is working as per the expected behaviour, and there is nothing to fix in the script, if the GET Rest call passes within the given attempts, the test automatically passes.

    One does not really need to change the test case status, if the test step fails, test case automatically fails. If there is no failed step means, test case passes automatically. Not sure the need for what you are asking.

     

    • towardsQuality's avatar
      towardsQuality
      Occasional Contributor
      11 months ago

      Thanks, nmrao! I really appreciate all your help.

      I do understand what the script is doing. The only updates I made were to set maxAttempts = 2 and to comment out the doubleIt lines:

      //def doubleIt = { it * 2 }

      //currentWait = (it == 1) ? currentWait : doubleIt(currentWait)

      There is a lot of setup in the test and regardless of whether this groovy step passes or fails, I need to test steps to continue so that the cleanup steps can remove the data from the database.

      The 'assert false' is throwing the java error I mentioned and completely stopping the test.

      In researching, I tried the following, which doesn't throw the java error, but it also stops the test flow (doesn't proceed to next test step):

      //if (item == maxAttempts) {assert false, "GET instance failed after $item attempts"}
      if (item == maxAttempts) {testRunner.fail("GET instance failed after $item attempts")}

      NOTE: under preferences -> Default Test Case Options -> Abort on Error is UNchecked for 'Abort test if an error occurs'.

      Aside: my last question about changing the test case status is not for this scenario, but for future reference. I have all the code I need to check all the test steps to see if they ended up as passed or failed. It would still be nice to be able to set the test case to pass using groovy if need be.

      • nmrao's avatar
        nmrao
        Champion Level 3
        11 months ago
        towardsQuality wrote:

        I do understand what the script is doing. The only updates I made were to set maxAttempts = 2 and to comment out the doubleIt lines:

        Not surprising from the error / Got it from the message. Quick way to test by reducing the attempts.

        towardsQuality wrote:

        The 'assert false' is throwing the java error I mentioned and completely stopping the test.

        That is desired functionality. This way, test step fails, so is the test case.

        towardsQuality wrote:

        //def doubleIt = { it * 2 } //currentWait = (it == 1) ? currentWait : doubleIt(currentWait)

        Up to you. Since the GET test step  is failing, no point in hitting every x seconds, so i choose different type of delay before retrying so that traffic would be less.

        towardsQuality wrote:

        NOTE: under preferences -> Default Test Case Options -> Abort on Error is UNchecked for 'Abort test if an error occurs'.

        This was my very first response to your question suggesting to look at the documentation.

        towardsQuality wrote:

        Aside: my last question about changing the test case status is not for this scenario, but for future reference. I have all the code I need to check all the test steps to see if they ended up as passed or failed. It would still be nice to be able to set the test case to pass using groovy if need be.

        I have already clarified in my previous response on the need of test case status.

        Test case status is nothing  but the collection of its test steps status. There is no point in passing the test case if one of the test steps is failing or vice versa. Test case status is implicitly updated based on the status of the steps, not explicit. It is designed that way, IMO.

        Or provide a justifiable use case for the same, a feature request can be made with Smartbear.