Forum Discussion

rajat_wrk's avatar
rajat_wrk
Occasional Contributor
4 years ago

Groovy Rerun Failed Test Case Execution Report

I am using groovy script to rerun failed test cases but in Junit-Style Report , the test cases are still coming as failed even though they got passed after several retries.    Could you please let ...
Data-Driven Testing
REST
Scripting
  • nmrao's avatar
    nmrao
    4 years ago
    rajat_wrk wrote:

     

    The point here is not the execution but the reporting, after multiple retries if the Test Cases is passed it is  marked  as Pass at Test Suit level on the tool interface but in the Test Suit Printable Report is still shows fail.

    It is being addressed the issue incorrectly. First look for the reason why it is happening. If the services works with definitive behavior, then there is no issue of reporting.

     

    rajat_wrk wrote:

    I have added assertion as a Test step in the test case. Since, the assertion can pass only if the value in response matches the assertion value, the test case may fail on multiple retries but it will keep on re running for a fixed number of times.

    Please check with your team why such indefinitve behavior?

    How will the end user / customer use it if one can't predict the expected response based on the input?

     

    For each fixed input, the response should always be predictable. And respective assertions must always pass. Otherwise, I am afraid the application behavior can be acceptable.

rajat_wrk's avatar
rajat_wrk
Occasional Contributor

nmrao  sorry for the confusion.

 

Here's the  teardown script at project level I am using:

 

def testSuiteResults = context.getTestRunner().getResults()
def testCaseResults
def runCount =0
def testPassed = true

// Iterate through each test case in the project
testSuiteResults.each {
it.getResults().each {
// PASSED test case
if ( it.getStatus().toString().equals("PASS") ) {
log.info "The test case '" + it.getTestCase().getName() + "' has passed!"
// FAILED test case
} else {
testPassed = false
log.info "The test case '" + it.getTestCase().getName() + "' has failed!"
runCount =0
// Rerunning the failed test case...
while ((runCount < 40) && (testPassed == false)) {
log.info "Rerunning the test for the " +runCount +"time"
if (it.getTestCase().run(new com.eviware.soapui.support.types.StringToObjectMap(), false).getStatus().toString().equals("PASS")) {
testPassed = true
log.info "The test case "+ it.getTestCase().getName() + " case has eventually passed!"
} else {
log.info "The test case "+ it.getTestCase().getName() + " did not pass"
}
runCount++
}
}
}
}

 

Using this script  I am able the  re-run failed test cases for maximum 40 times and test case may pass eventually. The issue is Junit-style report generated from tool still shows them as failed even if they pass after several retries.

Is there a way to keep the final result in the report?

 

nmrao's avatar
nmrao
Champion Level 3
4 years ago
Sorry, don't have any ideas in this case, unless you adopt some kind of program to achieve what you need.
However, I would do this:
1. No one will run the tests for 40 times to make them pass or run untill they pass. Look for the root cause such as
a. is it related to enviroment?
b. is it related to application issue?
c. any latencies between the test machines and the application?
d. any test dependencies? or data shared between them?

2. Irrespective of the above, try to run all the tests once and generate report in single short.
  • rajat_wrk's avatar
    rajat_wrk
    Occasional Contributor
    4 years ago

    Actually, the problem statement for me is that the api would return different response for consecutive requests and I need to validate a specific value whose order is random. So I am running it for maximum 40 times in case of failure.

     

    Is there any way to create a customized report using groovy which can keep the latest pass fail results. 

    • nmrao's avatar
      nmrao
      Champion Level 3
      4 years ago
      rajat_wrk wrote:

      Actually, the problem statement for me is that the api would return different response for consecutive requests

      How  are the assertions done in this case? Verifying them manually? If so, It requires to re-look at the tests and see the possibility to automate the behavior and increase the coverage of the tests accordingly.

      • rajat_wrk's avatar
        rajat_wrk
        Occasional Contributor
        4 years ago

        I have added assertion as a Test step in the test case. Since, the assertion can pass only if the value in response matches the assertion value, the test case may fail on multiple retries but it will keep on re running for a fixed number of times.

         

        The point here is not the execution but the reporting, after multiple retries if the Test Cases is passed it is  marked  as Pass at Test Suit level on the tool interface but in the Test Suit Printable Report is still shows fail.

        Is there a way to pick the final status and not the first instance in report.

         

        PS : I found a way to keep the results in csv file , it shows all the retries with execution status.