Use of tags in command line via bamboo build - Run all tests while sending -T tags parameters
Question In our CI build pipeline (bamboo) I setup two parameters tagTestSuite and tagTestCase. This way, all our automation testers can specify, if needed, which test cases need to be ran based on certain tags. This works fine if there are indeed such tags passed on. But what if a project just wants to run all of its tests? These -T parameters cannot be left empty in the command line as that fails the build... Deleting them from the bamboo variables will then lose the standardized approach for all testers to do a tag-based execution. Seeing the documentation here (https://support.smartbear.com/readyapi/docs/functional/running/automating/cli.html) I noticed you can provide logical operators. So I thought I came up with a nice solution: I provide as default value for those parameters in bamboo a tag value that nobody would use and by setting the "!" operator in front of it I'd expected that ALL test suites/cases would run. So I've set parameters and resulted in a command line that has these : "-TTestSuite !itShouldRunAllTestSuites" "-TTestCase !itShouldRunAllTestCase". Since none of the testSuite/testCases has a tag "itShouldRunAllTestSuites" or "itShouldRunAllTestCases" I thought my project would be run in full.... But nope: error is shown: [errorlog] java.lang.Exception: The tag "itShouldRunAllTestSuites" was not found. Yes, I could go and add a tag to every single test suite and test case to work around this, but that could be tedious and not so future proof... Answer Solution: ReadyAPI seems to look first for a tag and then complain it wasn't found. So I tried this: I've added the tags "itShouldRunAllTestCases" and "itShouldRunAllTestSuites" to my project but did not tag any of my testsuites/testcases with this tag. Executed above again and now ALL of my tests get's executed nicely!Gmail OAuth 2.0 API Automation Example and example SubmitListener.beforeSubmit
Hi. Recently I had a use case where I had to verify that our application sends an email in a particular case. In my case, it was an email whenever a certain status is reached, but it could also be for instance the sending of an invitation (user-creation) email or whatnot. In essence: I want to check at a give moment when the status condition is reached that the email is delivered to the proper email address. In the past, there we some free email generators (like mailinator, 10minutemail,...) where you could get API calls to check the inbox, but I couldn't find any that offered that feature for free. So I decided to setup a dedicated gmail email address for my tests and talk to the gmail api to read the inbox (all messages), verify the subject and the actual content of a mail and delete the messages. More info on the gmail api here: https://developers.google.com/gmail/api/reference/rest. To get started, I created a gmail user and setup an OAuth2.0 google client Id (https://console.cloud.google.com/apis/credentials). These settings (clientId and secret) I used to setup an Authorization code grant as described here: https://support.smartbear.com/readyapi/docs/requests/auth/types/oauth2/grants/auth-code.html So far so good, the readyAPI internal browers showed me the google popup and I could manually insert the authentication that was needed to generate successfully an access token from google. BUT: When I tried to automate the flow in this popup using the example code provided in the documentation (= https://support.smartbear.com/readyapi/docs/requests/auth/types/oauth2/automate/sample.html) it did not work for me. Therefore I wanted to share the changes I made to it in order to get it working. Also the event handler SubmitListener.beforeSubmit (described on the same page) needed some rework for me to work properly. As an extra, I also have a test case setup script that deletes all emails in the gmail inbox so I have a proper starting situation for my tests. Hope this can help any other testers that would need this! Automation Scripts tab of the Auth Manager I have encrypted project properties that store my gmail username (gmailUser) and password (gmailPass). Page 1: // This function asks for permission to use OAuth. The user must be logged in to use it. Logging in is performed in the script below. function consent() { if (document.getElementById('submit_approve_access')){ document.getElementById('submit_approve_access').click(); } } // This function fills user password in when the user name is already known. It uses the project-level "pass" property. function fillpwd() { document.getElementsByName('password')[0].value = '${#Project#gmailPass}'; document.getElementById('passwordNext').click(); window.setInterval(consent, 1000); } // This script checks what page is displayed and provides the appropriate data. It uses the project-level "user" and "pass" properties. if (document.getElementById('profileIdentifier')) { document.getElementById('profileIdentifier').click(); window.setTimeout(fillpwd, 1000) }else if (document.getElementById('identifierId') && document.getElementById('identifierNext')) { document.getElementById('identifierId').value = '${#Project#gmailUser}'; document.getElementById('identifierNext').click(); window.setTimeout(fillpwd, 1000); } else if (document.getElementByType('password')) { fillpwd(); } else if(document.getElementById('submit_approve_access')){ window.setInterval(consent, 100); } Page 2: function consent() { if (document.getElementById('submit_approve_access')){ document.getElementById('submit_approve_access').click(); } } window.setInterval(consent, 100); Event handler SubmitListener.beforeSubmit: Note: There might be some redundant iteration of code in there, feel free to rewrite, main thing is: it works. I also expected that I could use the "Target" column to filter on the requests steps that start with "gmail*" but that didn't do it. So I fixed that in another way, together with providing some smart checking whether a new token generation is needed or not (gmail token is valid for 60 minutes). // Import the required classes import com.eviware.soapui.impl.rest.actions.oauth.OltuOAuth2ClientFacade; import com.eviware.soapui.support.editor.inspectors.auth.TokenType; import com.eviware.soapui.model.support.ModelSupport; import java.time.LocalDateTime import java.time.format.DateTimeFormatter def testStepName = context.getModelItem().getName() def expiresOn = context.expand('${#Project#expiresOn}') if (testStepName.toLowerCase().contains("gmail")) { // IF expiresOn == "" OR dateNow is > expiresOn then we need to get a new token. Otherwise the old should still do.... TimeZone.setDefault(TimeZone.getTimeZone('UTC')) TimeZone tz = TimeZone.getTimeZone("UTC") LocalDateTime dateNow = LocalDateTime.now() def patternUTC = "yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'" DateTimeFormatter dateFormatUTC = DateTimeFormatter.ofPattern(patternUTC).withLocale(Locale.US) String nowUtcFormat = dateNow.format(dateFormatUTC) LocalDateTime dateExpiresOn = dateNow.plusMinutes(55) String expiresOnUtcFormat = dateExpiresOn.format(dateFormatUTC) if (expiresOn == "") { /* log.info("Expires on is empty, so we need to run submitListener and get new token. We also write the new expiresOn to the project properties!") log.info("nowUtcFormat = " + nowUtcFormat) log.info("dateExpiresOn = " + expiresOnUtcFormat) log.info "let's run the submitListener.beforeSubmit to get a new accessToken. We know this one will be valid for 60 minutes, so we set a the expiresOn project property to now + 55 minutes" */ // Set up variables def project = ModelSupport.getModelItemProject(context.getModelItem()) project.setPropertyValue("expiresOn", expiresOnUtcFormat) def authProfile = project.getAuthRepository().getEntry("google") def oldToken = authProfile.getAccessToken() def tokenType = TokenType.ACCESS // Create a facade object def oAuthFacade = new OltuOAuth2ClientFacade(tokenType) // Request an access token in headless mode oAuthFacade.requestAccessToken(authProfile, true, true) // Wait until the access token gets updated int iteration = 0 while (oldToken == authProfile.getAccessToken() && iteration < 10) { sleep(500) iteration++ } // Post the info to the log log.info("Gmail authentication event handler: Project property \"expiresOn\" is empty! We get/set new token: " + authProfile.getAccessToken() + " with new expiresOn = " + expiresOnUtcFormat + ". The old token (with unknown expiresOn) was = " + oldToken) } else { def potentialNewExpiresOn = expiresOnUtcFormat //expiresOn retrieved from project properties dateExpiresOn = LocalDateTime.parse(expiresOn, dateFormatUTC) expiresOnUtcFormat = dateExpiresOn.format(dateFormatUTC) //log.info("dateExpiresOn = " + expiresOnUtcFormat) if (dateNow > dateExpiresOn) { // log.info "Expired! Let's get a new token..." // Set up variables def project = ModelSupport.getModelItemProject(context.getModelItem()) def authProfile = project.getAuthRepository().getEntry("google") def oldToken = authProfile.getAccessToken() def tokenType = TokenType.ACCESS // Create a facade object def oAuthFacade = new OltuOAuth2ClientFacade(tokenType) // Request an access token in headless mode oAuthFacade.requestAccessToken(authProfile, true, true) // Wait until the access token gets updated int iteration = 0 while (oldToken == authProfile.getAccessToken() && iteration < 10) { sleep(500) iteration++ } // Post the info to the log project.setPropertyValue("expiresOn", potentialNewExpiresOn) log.info("Token was expired! We get/set new token: " + authProfile.getAccessToken() + " with new expiresOn = " + potentialNewExpiresOn + ". The old token (expiresOn = $expiresOn vs now " + nowUtcFormat + ") was = " + oldToken) } else { //log.info "Not yet expired. Let's keep using the same old token (expiresOn = $expiresOn vs now " + nowUtcFormat + ")" } } } Setup script to delete all emails in the gmail inbox for proper start situation: I have a disabled test suite "WorkItem" with a test case named "GmailStartSituationCleanup" This test case has 4 steps: 1° GET gmail messagesList 2° Script "IterateOverAllGmailMessageIds" 3° DELETE gmail messageId 4° GET gmail messagesList-EmptyListCheck def testSuiteWorkItem = testRunner.testCase.testSuite.project.getTestSuiteByName("WorkItem") def testCaseGmailStartSituationCleanup = testSuiteWorkItem.getTestCaseByName("GmailStartSituationCleanup") testCaseGmailStartSituationCleanup.run(new com.eviware.soapui.support.types.StringToObjectMap(), false) The groovy test step 2° IterateOverAllGmailMessageIds = import com.eviware.soapui.support.JsonUtil def testStepAllMessages = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()-1) def teststepNameAllMessages = testStepAllMessages.getName() def testStepDeleteMessage = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()+1) testStepDeleteMessage.setDisabled(true) def testStepVerifAllDeleted = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()+2) testStepVerifAllDeleted.setDisabled(false) def responseStatus = testStepAllMessages.testRequest.response.responseHeaders["#status#"][0] if (responseStatus.contains("HTTP/1.1 2")){ def responseMessages = context.expand( '${'+teststepNameAllMessages+'#Response#$[\'messages\']}' ) if (responseMessages!= "" && responseMessages!= null){ def numberOfMessages = (JsonUtil.parseTrimmedText(responseMessages)).size() def id for (i=0;i<numberOfMessages;i++){ id = context.expand( '${'+teststepNameAllMessages+'#Response#$[\'messages\']['+i+'][\'id\']}' ) testStepDeleteMessage.setPropertyValue("messageId", id) log.info "Cleanup of gmail messages : Message "+(i+1).toString()+"/"+numberOfMessages.toString()+" with messageId $id will be deleted so we can continue with a proper starting situation..." testStepDeleteMessage.run(testRunner, context) } }else{ log.info "Cleanup of gmail messages : No messages found. We can continue with proper starting situation" testStepVerifAllDeleted.setDisabled(true) } }
Automate Environment creation for REST and SOAP services
This came up in a previous thread, and I've finally worked out how to populate environments into a project and how to set their endpoints accordingly. This is relatively simple and can be extended for adding multiple environments. We're currently using this to convert all of our various ReadyAPI / SoapUI NG Pro projects to support environments. It's working like a charm! // For REST Services import com.eviware.soapui.model.environment.* import com.eviware.soapui.config.* def project = context.testCase.testSuite.project; // Assuming binding is the first interface on the project. def binding = context.testCase.testSuite.project.getInterfaceAt(1).getName().toString(); // Getting current list of environment names def environNames = project.getEnvironmentNames().toString(); // Names of environments to add def name = "env1" // Environment URLs, number should match number in previous array for environment names project.addNewEnvironment(name); project.setActiveEnvironment(name); def environ = project.getActiveEnvironment(); def newService; // For REST service: newService = environ.addNewService(binding, com.eviware.soapui.config.ServiceConfig.Type.REST); def serviceConfig = newService.getConfig(); def endpointConfig; endpointConfig = serviceConfig.addNewEndpoint(); newService.buildEndpoint(endpointConfig); def isCopy1 = true; log.info(isCopy1); environ.populateService(newService, isCopy1); environ.release(); For SOAP services: import com.eviware.soapui.model.environment.* import com.eviware.soapui.config.* def project = context.testCase.testSuite.project; // Assuming binding is the first interface on the project. def binding = context.testCase.testSuite.project.getInterfaceAt(1).getName().toString(); // Getting current list of environment names def environNames = project.getEnvironmentNames().toString(); // Names of environments to add def name = "env1" // Environment URLs, number should match number in previous array for environment names project.addNewEnvironment(name); project.setActiveEnvironment(name); def environ = project.getActiveEnvironment(); def newService; // For SOAP service: newService = environ.addNewService(binding, com.eviware.soapui.config.ServiceConfig.Type.SOAP); // For REST service: // newService = environ.addNewService(binding, com.eviware.soapui.config.ServiceConfig.Type.REST); def serviceConfig = newService.getConfig(); def endpointConfig; endpointConfig = serviceConfig.addNewEndpoint(); endpointConfig.setStringValue(url) newService.buildEndpoint(endpointConfig); def isCopy1 = true; log.info(isCopy1); environ.populateService(newService, isCopy1); environ.release(); The two are almost identical except for a couple of lines. Other than that, these have both worked for me without any problems.Automated testing with Jenkins and bypassing Tests
I recently started working with my DevOps team to automate our ReadyAPI projects in Jenkins so we can automate some parts of our tests instead of having to manually re-check defects as they are fixed. The intent is to eventually end up automating a lot more but this was a great start. The problem I ran into was that, with the way our projects are stored in GitLab, there are some tests that may fail (rightfully so) depending on which environment was last tested against and where the automated test is running. My original plan was to use Tags to control what tests to run in Jenkins, but you can only specify which tags to run and not which ones to exclude. I did a bit more research and came up with a solution. I can likely improve on it with events now that I have started researching that, but I wanted to share. I set a Project level property called "Automated" and set it to false. This can be overridden with the testrunner.bat/testrunner.sh with a command line argument of "-PAutomated=true", so that the value is only ever true when it runs on Jenkins and the appropriate argument. With that, I put a groovy script test step at the beginning of the tests that would parse the value of Automated, and disable certain tests if the value was true, and re-enable them if the value was false. Example script here: def automated = context.expand( '${#Project#Automated}' ).toBoolean(); if (automated == false) { context.testCase.testSuite.testCases["TestCase1 to D isable"].setDisabled(false); context.testCase.testSuite.testCases["TestCase2 to D isable"].setDisabled(false); } else { context.testCase.testSuite.testCases["TestCase1 to Enable"].setDisabled(true); context.testCase.testSuite.testCases["TestCase2 to Enable"].setDisabled(true); }Properties in SoapUI - case study
The possibility to use properties is a very important aspect in SoapUI automation. And although they are widely used they tend to be not very well understood. Indeed the name 'property' seems a little unfit for an element that (by all means) is presented by SmartBear as a variable to hold certain values for later use in the test execution cycle. For instance, one can have a 'sessionId' element to hold the current session string or one can have a 'cookie' element to hold some cookie value and so on. 1. Defining properties Properties can be defined at the project level, at the test suite level, at the test case level or in a special properties step. For instance, in order to define or edit a property at the test case level one must click on the desired test case in the left hand navigator and then just select the 'Custom Properties' tab. This is similar for any test suite or project. As said before there is also the possibility to use a special 'properties step'. This step must be created inside a certain test case thus its properties will only be available inside that particular test case. 2. Reading properties 2.1 Referencing in SoapUI steps When wanting to reference a property, attention has to be paid to the level at which it was defined. There are different syntaxes that need to be written depending on it as presented in the table below: Level at which it is defined Syntax Example property name Example request Project level ${#Project#Property} Port http://myserver:${#Project#Port} Test suite level ${#TestSuite#Property} Port http://myserver:${#TestSuite#Port} Test case level ${#TestCase#Property} Port http://myserver:${#TestCase#Port} Special properties step ${#Property} Port http://myserver:${#Port} Following the above example, if one would have a project level property named Port with the value 8080, then that property could be used anywhere in the project by simply referencing it like ${#Project#Port}. Example: When sending the request, the value of the ${#Project#Port} would be replaced by its value (in our example 8080). 2.2 Referencing in scripts In order to read a property in a script the following syntax is required: /** * Notes: * There are 4 properties defined at 4 levels: project, test suite, test case and special properties step level * The special properties step is simply called 'Properties' */ def stepProperty = testRunner.testCase.getTestStepByName( "Properties" ).getPropertyValue( "myCustomProperty" ) def testCaseProperty = testRunner.testCase.getPropertyValue( "myCustomProperty" ) def testSuiteProperty = testRunner.testCase.testSuite.getPropertyValue( "myCustomProperty" ) def projectProperty = testRunner.testCase.testSuite.project.getPropertyValue( "myCustomProperty" ) 3. Updating properties Updating tests can be done using a groovy script or a special transfer property step. Both of the methods are simple and straightforward but for the purpuse of this article only a groovy script example will be shown: /** * Notes: * This script changes 4 properties defined at 4 levels: project, test suite, test case and special properties step level * The special properties step is simply called 'Properties' */ testRunner.testCase.getTestStepByName( "Properties" ).setPropertyValue( "myCustomProperty", "newSpecialValue" ) testRunner.testCase.setPropertyValue( "myCustomProperty", "newTestCaseValue" ) testRunner.testCase.testSuite.setPropertyValue( "myCustomProperty", "newTestSuiteValue" ) testRunner.testCase.testSuite.project.setPropertyValue( "myCustomProperty", "newProjectValue" )
