How to loop Test Suites or Test Cases

[jkrier]

I have a Project that is currently built as

Project

Test Suite

Test Case

Test Steps

I have many Test Steps at the top that run queries and get log in credentials and then the remaining Test Steps run 3 methods that take many different types of input that I need to loop through with the query results. At the bottom of the Test Step stack I have Data Source Loops.

What I would like is for it to be

Project

Test Suite

Test Case

Test Steps

Test Case

Test Steps

etc...

Or

Project

Test Suite

Test Case

Test Steps

Test Suite

Test Case

Test Steps

etc...

The reason I want the individual method Test Steps to be their own Test Case/ Test Suite is because the Project will be ran nightly in Jenkins and with the current model if one step fails the Pass/Fail percentage is 0% for the entire Project. This is not really what is happening since each Step is actually a separate Test Case.

Any help is appreciated

[Radford · Accepted solution]

I think you are correct when you say "individual method Test Steps to be their own Test Case", I've always approached things assuming that a test case should be a self contained unit that can be run independently, testing one specific thing.

When faced with an issue like yours, I've always created a "Common Library" Test Suite, where I create all of my reusable "common" test cases, and then in my actual test cases, use the Run TestCase Test Step to call these common test cases. These "common" test cases can return properties as required.

This structure I am suggesting would look something like:

Project

|

|--Test Suite 1

|  |--Test Case 1

|  |--Test Case 2

|

|--Test Suite 2

|  |--Test Case 1

|  |--Test Case 2

|

|--Test Suite Common Library

   |--Test Case To Do Set up Things

Where all of the test cases in Test Suite 1 and 2 would call the common library test case "Test Case To Do Set up Things".

Finally I alway disable my "Common Library" test suite so it is never "run" as a stand alone test suite, but you can still quite happily call its test cases.

As a side note your question made me think about the TestCase options? In particular the two options:

• Abort on Error
• Fail Test Case on Error

Have you investigated these to see if they help you?.

[jkrier]

Hi @Radford, thanks for replying.

I like your suggestion of having a common Test Suite that can be used by the Test Cases. I have been giving this some thought and I think I know how I can do this.

As for the Abort/Fail on error; I am not sure how I could use that since I don't want to abort the run if one step fails. The remaining tests would not be ran and so the entire "test" would be incomplete. I do have it set to "fail on error" but that is why if any iteration of the methods returns an error the entire test fails at 0% success. Unless there is some other way to use those options that I am unaware of I don't think they can help. I am always open to suggestions.

[Radford]

When I mentioned the test case options it really was an after thought as quite a few times I've found that people don't know they are there. If you have already spotted and investigated them I don't think there is anything I can add. The main point I really wanted to get across was the use of a "Common Library" test suite.

One thing I will add, is that some times I've found devloping/debugging common test cases a little bit fiddley in particular with the details returned (especially if you have common test cases asserting results). So I always paste the following code into each common test case tear down script:

def logPrefix = testCase.getName() + ': '

for(result in testRunner.getResults()){
	if(result.getStatus().toString() != 'OK' ){
		def failedTestStepName = result.getTestStep().getName()
		def logPrefixStep = logPrefix.trim() + failedTestStepName + ': '
		
		log.error(logPrefixStep + 'TestStep "' + failedTestStepName + '" finished with the status ' + result.getStatus().toString())

		for(testProperty in testCase.getTestStepByName(failedTestStepName).getPropertyList()){
			if(testProperty.isReadOnly()){
				log.info(logPrefixStep + 'Output property: ' + testProperty.getName() + ' = ' + testProperty.getValue())
			}else{
				log.info(logPrefixStep + 'Input property: ' + testProperty.getName() + ' = ' + testProperty.getValue())
			}
		}
		
		for(message in result.getMessages()){
			log.error(logPrefixStep + 'Error message: ' + message) 
		}
	}
}

Thus on errors the test case fails (as defined by the test case option) and all the relavent details are written to the log, in particular the input properties, allowing you then to run the common test case as a stand alone test case to investigate issues.

[jkrier]

So after your initial advice this is what I ended up doing. I broke my project up so it looks like this

Project

TestSuite

TestCase(common steps)

queries

scripts

test case runner

TestCase

TestCase

etc.

I am using a groovy script for each test case that gathers the required inputs from the queries and runs the "Run Test Case" step with a loop for however many combinations of inputs are available for that given method (Test Case).

This is an example of one of my "groovy script runners"

def stepIndex = testRunner.testCase.getTestStepIndexByName( context.currentStep.label )
def runStep = testRunner.testCase.getTestStepAt( stepIndex + 1 ).name
log.info runStep
def groovyUtils = new com.eviware.soapui.support.GroovyUtils(context)
def xmlResponseHolder = groovyUtils.getXmlHolder( "queryPORTAL_ORGANIZATION#ResponseAsXML" )
def responseAsXml = context.expand( '${queryPORTAL_ORGANIZATION#ResponseAsXml#//Results[1]/ResultSet[1]}' )
log.info "queryPORTAL_ORGANIZATIONResponseAsXml ==> " + responseAsXml
def int rowCount = context.expand( '${queryPORTAL_ORGANIZATION#ResponseAsXml#count(//Row)}' ).toInteger()
log.info "The Row Count Is: " + rowCount
for ( i = 1; i > rowCount; i++ ){
 def porg = xmlResponseHolder[ "(//Results/ResultSet/Row[ " + i + " ]/PORTAL_ORGANIZATION)" ].toString()
 log.info "Portal Organization: " + i + " " + "[ " + porg + " ]"
 def ts = new java.text.SimpleDateFormat("yyyy-MM-dd-HH:mm:ss.SSS")
 def randomString = System.currentTimeMillis()
 testRunner.testCase.testSteps[ "inputValues" ].setPropertyValue( "PORTAL_ORGANIZATION", porg.toString())
 testRunner.testCase.testSteps[ "inputValues" ].setPropertyValue( "trackingkey", randomString.toString())
 testRunner.testCase.testSteps[ "inputValues" ].setPropertyValue( "timestamp", ts.toString())
 testRunner.runTestStepByName( runStep )
 
}

stepIndex = stepIndex + 2
testRunner.gotoStep( stepIndex )

Per your last response. Currently what I do for that is I have an Event (TestRunListener.afterStep) that logs the request/response and status.

def testStepName = context.currentStep.label
log.info testStepName + " REQUEST " + context.currentStep.getPropertyValue("RawRequest")
log.info testStepName + " RESPONSE " + context.currentStep.getPropertyValue("RawResponse")
log.info testStepName + " TestStep has status " + "[" + testStepResult.status + "]"

I plan on trying your script out though. Gathering the properties sounds smart to me.

Thanks for your all of your help.