Forum Discussion

jkrier's avatar
jkrier
Regular Contributor
8 years ago

How to loop Test Suites or Test Cases

I have a Project that is currently built as

Project

Test Suite

Test Case

Test Steps

 

I have many Test Steps at the top that run queries and get log in credentials and then the remaining Test Steps run 3 methods that take many different types of input that I need to loop through with the query results. At the bottom of the Test Step stack I have Data Source Loops.

 

What I would like is for it to be

Project

Test Suite

Test Case

Test Steps

Test Case

Test Steps

etc...

 

Or

 

Project

Test Suite

Test Case

Test Steps

Test Suite

Test Case

Test Steps

etc...

 

The reason I want the individual method Test Steps to be their own Test Case/ Test Suite is because the Project will be ran nightly in Jenkins and with the current model if one step fails the Pass/Fail percentage is 0% for the entire Project. This is not really what is happening since each Step is actually a separate Test Case.

 

Any help is appreciated

  • Radford's avatar
    Radford
    8 years ago

    I think you are correct when you say "individual method Test Steps to be their own Test Case", I've always approached things assuming that a test case should be a self contained unit that can be run independently, testing one specific thing.

     

    When faced with an issue like yours, I've always created a "Common Library" Test Suite, where I create all of my reusable "common" test cases, and then in my actual test cases, use the Run TestCase Test Step to call these common test cases. These "common" test cases can return properties as required.

     

    This structure I am suggesting would look something like:

     

    Project

    |

    |--Test Suite 1

    |  |--Test Case 1

    |  |--Test Case 2

    |

    |--Test Suite 2

    |  |--Test Case 1

    |  |--Test Case 2

    |

    |--Test Suite Common Library

       |--Test Case To Do Set up Things

     

    Where all of the test cases in Test Suite 1 and 2 would call the common library test case "Test Case To Do Set up Things".

     

    Finally I alway disable my "Common Library" test suite so it is never "run" as a stand alone test suite, but you can still quite happily call its test cases.

     

    As a side note your question made me think about the TestCase options? In particular the two options:

     

    • Abort on Error
    • Fail Test Case on Error

    Have you investigated these to see if they help you?.

  • Radford's avatar
    Radford
    Super Contributor
    8 years ago

    I think you are correct when you say "individual method Test Steps to be their own Test Case", I've always approached things assuming that a test case should be a self contained unit that can be run independently, testing one specific thing.

     

    When faced with an issue like yours, I've always created a "Common Library" Test Suite, where I create all of my reusable "common" test cases, and then in my actual test cases, use the Run TestCase Test Step to call these common test cases. These "common" test cases can return properties as required.

     

    This structure I am suggesting would look something like:

     

    Project

    |

    |--Test Suite 1

    |  |--Test Case 1

    |  |--Test Case 2

    |

    |--Test Suite 2

    |  |--Test Case 1

    |  |--Test Case 2

    |

    |--Test Suite Common Library

       |--Test Case To Do Set up Things

     

    Where all of the test cases in Test Suite 1 and 2 would call the common library test case "Test Case To Do Set up Things".

     

    Finally I alway disable my "Common Library" test suite so it is never "run" as a stand alone test suite, but you can still quite happily call its test cases.

     

    As a side note your question made me think about the TestCase options? In particular the two options:

     

    • Abort on Error
    • Fail Test Case on Error

    Have you investigated these to see if they help you?.

    • jkrier's avatar
      jkrier
      Regular Contributor
      8 years ago

      Hi Radford, thanks for replying.

       

      I like your suggestion of having a common Test Suite that can be used by the Test Cases. I have been giving this some thought and I think I know how I can do this.

       

      As for the Abort/Fail on error; I am not sure how I could use that since I don't want to abort the run if one step fails. The remaining tests would not be ran and so the entire "test" would be incomplete. I do have it set to "fail on error" but that is why if any iteration of the methods returns an error the entire test fails at 0% success. Unless there is some other way to use those options that I am unaware of I don't think they can help. I am always open to suggestions.

      • Radford's avatar
        Radford
        Super Contributor
        8 years ago

        When I mentioned the test case options it really was an after thought as quite a few times I've found that people don't know they are there. If you have already spotted and investigated them I don't think there is anything I can add. The main point I really wanted to get across was the use of a "Common Library" test suite.

         

        One thing I will add, is that some times I've found devloping/debugging common test cases a little bit fiddley in particular with the details returned (especially if you have common test cases asserting results). So I always paste the following code into each common test case tear down script:

         

        def logPrefix = testCase.getName() + ': '

for(result in testRunner.getResults()){
	if(result.getStatus().toString() != 'OK' ){
		def failedTestStepName = result.getTestStep().getName()
		def logPrefixStep = logPrefix.trim() + failedTestStepName + ': '
		
		log.error(logPrefixStep + 'TestStep "' + failedTestStepName + '" finished with the status ' + result.getStatus().toString())

		for(testProperty in testCase.getTestStepByName(failedTestStepName).getPropertyList()){
			if(testProperty.isReadOnly()){
				log.info(logPrefixStep + 'Output property: ' + testProperty.getName() + ' = ' + testProperty.getValue())
			}else{
				log.info(logPrefixStep + 'Input property: ' + testProperty.getName() + ' = ' + testProperty.getValue())
			}
		}
		
		for(message in result.getMessages()){
			log.error(logPrefixStep + 'Error message: ' + message) 
		}
	}
}

         

        Thus on errors the test case fails (as defined by the test case option) and all the relavent details are written to the log, in particular the input properties, allowing you then to run the common test case as a stand alone test case to investigate issues.