Automated testing with Jenkins and bypassing Tests

I recently started working with my DevOps team to automate our ReadyAPI projects in Jenkins so we can automate some parts of our tests instead of having to manually re-check defects as they are fixed. The intent is to eventually end up automating a lot more but this was a great start. The problem I ran into was that, with the way our projects are stored in GitLab, there are some tests that may fail (rightfully so) depending on which environment was last tested against and where the automated test is running.

 

My original plan was to use Tags to control what tests to run in Jenkins, but you can only specify which tags to run and not which ones to exclude. I did a bit more research and came up with a solution. I can likely improve on it with events now that I have started researching that, but I wanted to share.

 

I set a Project level property called "Automated" and set it to false. This can be overridden with the testrunner.bat/testrunner.sh with a command line argument of "-PAutomated=true", so that the value is only ever true when it runs on Jenkins and the appropriate argument. With that, I put a groovy script test step at the beginning of the tests that would parse the value of Automated, and disable certain tests if the value was true, and re-enable them if the value was false. 

 

Example script here:

def automated = context.expand( '${#Project#Automated}' ).toBoolean();
if (automated == false)
{
	context.testCase.testSuite.testCases["TestCase1 to D isable"].setDisabled(false);
	context.testCase.testSuite.testCases["TestCase2 to D isable"].setDisabled(false);
}
else
{
	context.testCase.testSuite.testCases["TestCase1 to Enable"].setDisabled(true);
	context.testCase.testSuite.testCases["TestCase2 to Enable"].setDisabled(true);
}