Recent Content
ReadyAPI and TestEngine with Zephyr Scale
In this post, we are going to talk about using some of the SmartBear API testing tools and writing the results in an automated fashion to Zephyr Scale. But first let's take a step back and discuss a general approach of using testing tools with test management tools. At a manual level, when creating simple Multi-Protocol API tests, we might use a lighter weight tool like SoapUI Open Source, a great tool for quick and basic functional and performance API tests. After we trigger the test manually, we as humans would parse the report for any details on the test run, and to track it would update a spreadsheet, testcase in Jira, but somewhere. This sequence is important to note for when we start to advance to an automated workflow. Now let's talk about automating this process. We execute our API tests in an ephemeral environment, publish the results to a relative location, and then pass those results from that relative location to a test management platform. It is quite simple. In this workflow it tends to be easier to use a tool that has native integrations into CI platforms like Jenkins and Azure Dev Ops, that make managing our executions in these ephemeral environments much easier. Here, ReadyAPI, or TestEngine is the clear choice as they both seamlessly integrate into any CI-CD system and have native integrations into Jenkins, Azure, and many more. Now that sequence I mentioned earlier is very important. We will execute our API tests, and then pass the results into Zephyr Scale as step 2 in the pipeline. Here is a look at a scripted pipeline approach, note that there is freestyle job options available in Jenkins too. ReadyAPI Jenkins Plugin: https://support.smartbear.com/readyapi/docs/integrations/jenkins.html TestEngine Jenkins Plugin: https://support.smartbear.com/testengine/docs/admin/jenkins.html node { stage('Run API Tests') { // Run the API Tests using ReadyAPI or TestEngine stage('Pass Results') { //Pass Results to Zephyr Scale } } Now we need to be a little more specific with the 'Run API Tests' Stage. Above I mention when we run our API Tests in an automated fashion, we need to write the results to a relative location so that we can then send those results to Zephyr Scale from that location. Both ReadyAPI and TestEngine allow us to write results to locations as part of the command-line or native integrations. I will show command- line options for ReadyAPI and UI native integration for TestEngine but both options are available for both tools. Starting with ReadyAPI, testrunner CLI, the -f and -F flags represent the directory we are writing to, and the report format, respectively. ReadyAPI offers reports in PDF, XLS, HTML, RTF, CSV, TXT and XML, but the automation recommendation would be to pass results in the Junit-XML option. At a basic level we need this: testrunner.bat [optional-arguments] <test-project> And we need to specify -f and -F testrunner.bat -f<directory> -F<args> <test-project> and with -f requiring a relative directory, that can change based on the CI system. I will use Azure Dev Ops for both my examples here. In Azure I pull my test cases from the $(System.DefaultWorkingDirectory), which contains my git repo. In Azure I publish results to the $(Common.TestResultsDirectory) An example full command would look like this: "C:\Program Files\SmartBear\ReadyAPI-3.40.1\bin\testrunner.bat" -r -a -j -f$(Common.TestResultsDirectory) "-RJUnit-Style HTML Report" -FXML "-EDefault environment" "$(System.DefaultWorkingDirectory)/" With TestEngine it's very similar, but I am highlighting it through the native integration, note the publish test results and save Junit report option enabled below: Now lastly, we need to send the results to Zephyr Scale from the pipeline, before our release is over. I think it's easiest with the Zephyr Scale API: https://support.smartbear.com/zephyr-squad-cloud/docs/api/index.html along with the Auto-Create Test Case option to true. The command below is a basic example, and replica of the one seen in the Azure Pipeline screenshot. curl -H "Authorization: Bearer Zephyr-Scale-Token-Here" -F file= Relative-Location-of-Report-Here\report.xml;type=application/xml "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey=Project-Key-Here&autoCreateTestCases=true" After you modify the API token, Relative location, and project key you are good to run the pipeline. When we jump to Jira, we can see inside Zephyr Scale that the results are populating. Even with transactional Data on the failed test steps0likes0CommentsSimple script to batch edit multiple groovy scripts
Question How to batch edit multiple groovy scripts? Here is my solution. Answer import com.eviware.soapui.impl.wsdl.teststeps.WsdlGroovyScriptTestStep def testSuite = 'testsuitename' String textToSearch = """text""" def replacementText = """text""" def tSuite = context.testCase.testSuite.project.testSuites[testSuite] def testcases = tSuite.testCaseList.toArray() testcases.each() {Case -> for( testSteps in Case.testStepList ) { if( testSteps instanceof WsdlGroovyScriptTestStep && testSteps.getScript().contains(textToSearch)) { log.info(Case.getName().toString() + " - " + testSteps.getScript().toString()) String newString = testSteps.getScript().toString().replace(textToSearch, replacementText) testSteps.setScript(newString) } } }3likes0CommentsUse of tags in command line via bamboo build - Run all tests while sending -T tags parameters
Question In our CI build pipeline (bamboo) I setup two parameters tagTestSuite and tagTestCase. This way, all our automation testers can specify, if needed, which test cases need to be ran based on certain tags. This works fine if there are indeed such tags passed on. But what if a project just wants to run all of its tests? These -T parameters cannot be left empty in the command line as that fails the build... Deleting them from the bamboo variables will then lose the standardized approach for all testers to do a tag-based execution. Seeing the documentation here (https://support.smartbear.com/readyapi/docs/functional/running/automating/cli.html)I noticed you can provide logical operators. So I thought I came up with a nice solution: I provide as default value for those parameters in bamboo a tag value that nobody would use and by setting the "!" operator in front of it I'd expected thatALL test suites/cases would run. So I've set parameters and resulted in a command line that has these : "-TTestSuite !itShouldRunAllTestSuites" "-TTestCase !itShouldRunAllTestCase". Since none of the testSuite/testCases has a tag "itShouldRunAllTestSuites" or "itShouldRunAllTestCases" I thought my project would be run in full.... But nope: error is shown:[errorlog] java.lang.Exception: The tag "itShouldRunAllTestSuites" was not found. Yes, I could go and add a tag to every single test suite and test case to work around this, but that could be tedious and not so future proof... Answer Solution: ReadyAPI seems to look first for a tag and then complain it wasn't found. So I tried this: I've added the tags "itShouldRunAllTestCases" and "itShouldRunAllTestSuites" to my project but did not tag any of my testsuites/testcases with this tag. Executed above again and now ALL of my tests get's executed nicely!0likes0CommentsGmail OAuth 2.0 API Automation Example and example SubmitListener.beforeSubmit
Hi. Recently I had a use case where I had to verify that our application sends an email in a particular case. In my case, it was an email whenever a certain status is reached, but it could also be for instance the sending of an invitation (user-creation) email or whatnot. In essence: I want to check at a give moment when the status condition is reached that the email is delivered to the proper email address. In the past, there we some free email generators (like mailinator, 10minutemail,...) where you could get API calls to check the inbox, but I couldn't find any that offered that feature for free. So I decided to setup a dedicated gmail email address for my tests and talk to the gmail api to read the inbox (all messages), verify the subject and the actual content of a mail and delete the messages. More info on the gmail api here: https://developers.google.com/gmail/api/reference/rest. To get started, I created a gmail user and setup an OAuth2.0 google client Id (https://console.cloud.google.com/apis/credentials). These settings (clientId and secret) I used to setup an Authorization code grant as described here:https://support.smartbear.com/readyapi/docs/requests/auth/types/oauth2/grants/auth-code.html So far so good, the readyAPI internal browers showed me the google popup and I could manually insert the authentication that was needed to generate successfully an access token from google. BUT: When I tried to automate the flow in this popup using the example code provided in the documentation (=https://support.smartbear.com/readyapi/docs/requests/auth/types/oauth2/automate/sample.html) it did not work for me. Therefore I wanted to share the changes I made to it in order to get it working. Also the event handler SubmitListener.beforeSubmit(described on the same page) needed some rework for me to work properly. As an extra, I also have a test case setup script that deletes all emails in the gmail inbox so I have a proper starting situation for my tests. Hope this can help any other testers that would need this! Automation Scripts tab of the Auth Manager I have encrypted project properties that store my gmail username (gmailUser) and password (gmailPass). Page 1: // This function asks for permission to use OAuth. The user must be logged in to use it. Logging in is performed in the script below. function consent() { if (document.getElementById('submit_approve_access')){ document.getElementById('submit_approve_access').click(); } } // This function fills user password in when the user name is already known. It uses the project-level "pass" property. function fillpwd() { document.getElementsByName('password')[0].value = '${#Project#gmailPass}'; document.getElementById('passwordNext').click(); window.setInterval(consent, 1000); } // This script checks what page is displayed and provides the appropriate data. It uses the project-level "user" and "pass" properties. if (document.getElementById('profileIdentifier')) { document.getElementById('profileIdentifier').click(); window.setTimeout(fillpwd, 1000) }else if (document.getElementById('identifierId') && document.getElementById('identifierNext')) { document.getElementById('identifierId').value = '${#Project#gmailUser}'; document.getElementById('identifierNext').click(); window.setTimeout(fillpwd, 1000); } else if (document.getElementByType('password')) { fillpwd(); } else if(document.getElementById('submit_approve_access')){ window.setInterval(consent, 100); } Page 2: function consent() { if (document.getElementById('submit_approve_access')){ document.getElementById('submit_approve_access').click(); } } window.setInterval(consent, 100); Event handler SubmitListener.beforeSubmit: Note: There might be some redundant iteration of code in there, feel free to rewrite, main thing is: it works. I also expected that I could use the "Target" column to filter on the requests steps that start with "gmail*" but that didn't do it. So I fixed that in another way, together with providing some smart checking whether a new token generation is needed or not (gmail token is valid for 60 minutes). // Import the required classes import com.eviware.soapui.impl.rest.actions.oauth.OltuOAuth2ClientFacade; import com.eviware.soapui.support.editor.inspectors.auth.TokenType; import com.eviware.soapui.model.support.ModelSupport; import java.time.LocalDateTime import java.time.format.DateTimeFormatter def testStepName = context.getModelItem().getName() def expiresOn = context.expand('${#Project#expiresOn}') if (testStepName.toLowerCase().contains("gmail")) { // IF expiresOn == "" OR dateNow is > expiresOn then we need to get a new token. Otherwise the old should still do.... TimeZone.setDefault(TimeZone.getTimeZone('UTC')) TimeZone tz = TimeZone.getTimeZone("UTC") LocalDateTime dateNow = LocalDateTime.now() def patternUTC = "yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'" DateTimeFormatter dateFormatUTC = DateTimeFormatter.ofPattern(patternUTC).withLocale(Locale.US) String nowUtcFormat = dateNow.format(dateFormatUTC) LocalDateTime dateExpiresOn = dateNow.plusMinutes(55) String expiresOnUtcFormat = dateExpiresOn.format(dateFormatUTC) if (expiresOn == "") { /* log.info("Expires on is empty, so we need to run submitListener and get new token. We also write the new expiresOn to the project properties!") log.info("nowUtcFormat = " + nowUtcFormat) log.info("dateExpiresOn = " + expiresOnUtcFormat) log.info "let's run the submitListener.beforeSubmit to get a new accessToken. We know this one will be valid for 60 minutes, so we set a the expiresOn project property to now + 55 minutes" */ // Set up variables def project = ModelSupport.getModelItemProject(context.getModelItem()) project.setPropertyValue("expiresOn", expiresOnUtcFormat) def authProfile = project.getAuthRepository().getEntry("google") def oldToken = authProfile.getAccessToken() def tokenType = TokenType.ACCESS // Create a facade object def oAuthFacade = new OltuOAuth2ClientFacade(tokenType) // Request an access token in headless mode oAuthFacade.requestAccessToken(authProfile, true, true) // Wait until the access token gets updated int iteration = 0 while (oldToken == authProfile.getAccessToken() && iteration < 10) { sleep(500) iteration++ } // Post the info to the log log.info("Gmail authentication event handler: Project property \"expiresOn\" is empty! We get/set new token: " + authProfile.getAccessToken() + " with new expiresOn = " + expiresOnUtcFormat + ". The old token (with unknown expiresOn) was = " + oldToken) } else { def potentialNewExpiresOn = expiresOnUtcFormat //expiresOn retrieved from project properties dateExpiresOn = LocalDateTime.parse(expiresOn, dateFormatUTC) expiresOnUtcFormat = dateExpiresOn.format(dateFormatUTC) //log.info("dateExpiresOn = " + expiresOnUtcFormat) if (dateNow > dateExpiresOn) { // log.info "Expired! Let's get a new token..." // Set up variables def project = ModelSupport.getModelItemProject(context.getModelItem()) def authProfile = project.getAuthRepository().getEntry("google") def oldToken = authProfile.getAccessToken() def tokenType = TokenType.ACCESS // Create a facade object def oAuthFacade = new OltuOAuth2ClientFacade(tokenType) // Request an access token in headless mode oAuthFacade.requestAccessToken(authProfile, true, true) // Wait until the access token gets updated int iteration = 0 while (oldToken == authProfile.getAccessToken() && iteration < 10) { sleep(500) iteration++ } // Post the info to the log project.setPropertyValue("expiresOn", potentialNewExpiresOn) log.info("Token was expired! We get/set new token: " + authProfile.getAccessToken() + " with new expiresOn = " + potentialNewExpiresOn + ". The old token (expiresOn = $expiresOn vs now " + nowUtcFormat + ") was = " + oldToken) } else { //log.info "Not yet expired. Let's keep using the same old token (expiresOn = $expiresOn vs now " + nowUtcFormat + ")" } } } Setup script to delete all emails in the gmail inbox for proper start situation: I have a disabled test suite "WorkItem" with a test case named "GmailStartSituationCleanup" This test case has 4 steps: 1°GET gmail messagesList 2° Script "IterateOverAllGmailMessageIds" 3°DELETE gmail messageId 4°GET gmail messagesList-EmptyListCheck def testSuiteWorkItem = testRunner.testCase.testSuite.project.getTestSuiteByName("WorkItem") def testCaseGmailStartSituationCleanup = testSuiteWorkItem.getTestCaseByName("GmailStartSituationCleanup") testCaseGmailStartSituationCleanup.run(new com.eviware.soapui.support.types.StringToObjectMap(), false) The groovy test step 2°IterateOverAllGmailMessageIds = import com.eviware.soapui.support.JsonUtil def testStepAllMessages = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()-1) def teststepNameAllMessages = testStepAllMessages.getName() def testStepDeleteMessage = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()+1) testStepDeleteMessage.setDisabled(true) def testStepVerifAllDeleted = testRunner.testCase.getTestStepAt(context.getCurrentStepIndex()+2) testStepVerifAllDeleted.setDisabled(false) def responseStatus = testStepAllMessages.testRequest.response.responseHeaders["#status#"][0] if (responseStatus.contains("HTTP/1.1 2")){ def responseMessages = context.expand( '${'+teststepNameAllMessages+'#Response#$[\'messages\']}' ) if (responseMessages!= "" && responseMessages!= null){ def numberOfMessages = (JsonUtil.parseTrimmedText(responseMessages)).size() def id for (i=0;i<numberOfMessages;i++){ id = context.expand( '${'+teststepNameAllMessages+'#Response#$[\'messages\']['+i+'][\'id\']}' ) testStepDeleteMessage.setPropertyValue("messageId", id) log.info "Cleanup of gmail messages : Message "+(i+1).toString()+"/"+numberOfMessages.toString()+" with messageId $id will be deleted so we can continue with a proper starting situation..." testStepDeleteMessage.run(testRunner, context) } }else{ log.info "Cleanup of gmail messages : No messages found. We can continue with proper starting situation" testStepVerifAllDeleted.setDisabled(true) } }3likes0CommentsHow to capture teststep status as pass/fail using groovy to write it in txt file
Question How to capture teststep status as pass/fail using groovy to write it in txt file Answer This can be done very simply by using the following TearDown Script: // Define variables for holding test suites, test cases and test steps def testSuites def testCases def testSteps // Get all the test suites from the project testSuites = project.testSuiteList File file = new File("C:\\Users\\luciana\\Desktop\\test.txt") /** * Iterate through each test suite, test case and test step */ testSuites.each() { // Log test suite name file << "-----------------------------------\n" file << "Running test suite: " + it.getName() + "\n" file << "-----------------------------------\n" // Get a list with the contained test cases testCases = it.getTestCaseList() testCases.each() { // Log test case name file << "-----------------------------------\n" file << "Running test case: " + it.getName() + "\n" file << "-----------------------------------\n" // Get a list with the contained test steps testSteps = it.getTestStepList() testSteps.each() { file << it.getName() + " - " + it.getAssertionStatus() + "\n" } } }1like0CommentsGroovy Script to identify a file timestamp
Question How to identify a file timestamp? Answer Just sharing this groovy script I am using to disable a "DataCollection" step based on a filetime stamp. Use case: Identify when datasource file is modified If datasource file is already generated then skip/disable datacollection step which in the end writes data to datasource file why to disable - if you want to run the same API tests (but different versions say due to refactoring) against same dataset Below is the sample test case structure DataSource check script is shown below import com.eviware.soapui.support.GroovyUtils import java.text.DateFormat import java.text.SimpleDateFormat //## Get test step name // def currentStepInd = context.currentStepIndex def TestStepName = testRunner.testCase.getTestStepAt(currentStepInd).name log.info "------------------------------------------------" log.info "Running $TestStepName..." //## Get current Date ##// def CurrentDate = new Date() log.info "Current Date is $CurrentDate..." // Get datasource file // def DataSourceFile = context.expand('${projectDir}') + "\\DataSource.txt" File DataFile = new File(DataSourceFile) log.info "DataSource File is: $DataFile" def fileDate if (DataFile.exists()) { // Get the last modification information. Long lastModified = DataFile.lastModified() // Create a new date object and pass last modified fileDate = new Date(lastModified) //fileDate = sdf.format(fileDate) log.info "File modified time is: $fileDate" } //## To find Date Diff ##// def diff use(groovy.time.TimeCategory) { diff = (CurrentDate - fileDate).days } //## Skip DataCollection if DataSource is older than today ##// if(diff == 0) { //## disable teststep to skip data collection ##// log.info "Disabling testStep DataCollection..." testRunner.testCase.getTestStepByName( "DataCollection" ).setDisabled(true) }else{ //## enable teststep to run data collection ##// log.info "Enabling testStep DataCollection..." testRunner.testCase.getTestStepByName( "DataCollection" ).setDisabled(false) } log.info "Finished $TestStepName..." log.info "------------------------------------------------" thanks!0likes0CommentsHow to get cookies in Groovy scripts?
Question How to get cookies in Groovy scripts? Answer If theSession optionis enabled, ReadyAPI maintains the HTTP session on the TestCase level automatically, which means that it storescookiesreceived from the server and sends them back in the subsequent requests to the same endpoint. If you need to get programmatic access to the stored cookies, you can use this Groovy script (it's applicable to ReadyAPI v.1.7.0 and later): import org.apache.http.protocol.HttpContext import com.eviware.soapui.model.iface.SubmitContext import org.apache.http.impl.client.BasicCookieStore import org.apache.http.client.protocol.HttpClientContext HttpContext httpContext = context.getProperty(SubmitContext.HTTP_STATE_PROPERTY) BasicCookieStore cookieStore = httpContext.getAttribute(HttpClientContext.COOKIE_STORE) //iterate through the cookies store and output the name-value pairs to the log cookieStore.getCookies().each{ log.info(it.name + "=" + it.value) } Example:1like0CommentsHow to generate a random Number, String, AlphaNumeric string
Question How to generate a random Number, String, AlphaNumeric string Answer This below function will generate random Number, String, alphaNumeric string as what you pass in parameters. Refer below code and help yourself in generating random numbers. def num = generateRndString(10, "numeric"); log.info num def str = generateRndString(10, "string"); log.info str def alphaNum = generateRndString(10, "alphanumeric"); log.info alphaNum testRunner.testCase.getTestStepByName("Properties").setPropertyValue("RndNum", num) testRunner.testCase.getTestStepByName("Properties").setPropertyValue("RndString", str) testRunner.testCase.getTestStepByName("Properties").setPropertyValue("RndAlpha", alphaNum) def generateRndString(int num, String type){ def randValue = ""; if( type.equalsIgnoreCase("numeric") ){ def alphaNumeric = ('0'..'9').join() randValue = RandomStringUtils.random(num, alphaNumeric) while (randValue.size()!=num) { randValue = RandomStringUtils.random(num, alphaNumeric) } } else if( type.equalsIgnoreCase("string") ){ def alphaNumeric = (('a'..'z')+('A'..'Z')).join() randValue = RandomStringUtils.random(num, alphaNumeric) while (randValue.size()!=num) { randValue = RandomStringUtils.random(num, alphaNumeric) } } else if( type.equalsIgnoreCase("alphanumeric") ){ def alphaNumeric = (('0'..'9')+('a'..'z')+('A'..'Z')).join() randValue = RandomStringUtils.random(num, alphaNumeric) while (randValue.size()!=num) { randValue = RandomStringUtils.random(num, alphaNumeric) } } return randValue }0likes0CommentsGroovy Automated DataSource Loopers
Symptoms You may need to create the loop steps through groovy manually.So without further ado this is how I script my groovy loopers NOTE: in this example I am connecting to an OracleDB to fuel my requests. Solution 1) Structure: The structure within my test case is quite simple:loopStarterto connect to the db/initialise the data - storing the current information into properties,responseto send these properties off to the API through a request,loopEnderto either propagate or exit the loop if the condition is met. 2) loopStarter Connection to/creation of your data source, be it a database, excel etc In this example I will be using the groovy.sqlclass (documentation) to connect to my db, and store the results into properties import groovy.sql.Sql sql = Sql.newInstance(<connection details>) def res = sql.rows(<sql query>) def loopProperties = testRunner.testCase.getTestStepByName("loopProperties") //will initialise count in the loopProperties step if count does not yet exist if(!loopProperties.hasProperty("count")){ loopProperties.setPropertyValue("count","0") } def count = Integer.parseInt(loopProperties.getPropertyValue("count")) //store the properties from the current result loopProperties.setPropertyValue("x",res[count].x) loopProperties.setPropertyValue("y",res[count].y) loopProperties.setPropertyValue("querySize",(String)res.size()) sql.close() sql.rows returns an array of ArrayLists, so to access the current result (more on this later) we will use res[count]. Just to reiterate this step ONLY sets the properties to be sent off in the request.querySize is set so we can continue looping over all the results from the query. 3) response To then call your properties into your request step we will do this: <soapenv:Envelope namespace:ns="namespace"> <soapenv:Header/> <soapenv:Body> <ns:x>${loopProperties#x}</ns:x> <ns:y>${loopProperties#y}</ns:y> </soapenv:Body> </soapenv:Envelope> if you choose not to store your properties in a property step and wanted to store them at the test case level you can fetch them like so: ... <ns:x>${#TestCase#x}</ns:x> ... 4) loopEnder This is the loop exit condition. It will tell the loop whether or not there are more results from your query to send through the request. This is also a pretty simple step: - check count vs querySize - increment count if true - go to loopStarter if true, to update the properties (count has been increased andbecause of this, so will the current result) def loopProperties = testRunner.testCase.getTestStepByName("loopProperties") def count = Integer.parseInt(loopProperties.getPropertyValue("count")) def querySize = Integer.parseInt(loopProperties.getPropertyValue("querySize")) if( count<querySize-1 ){ count = count+1 loopProperties.setPropertyValue("count", (String)count) testRunner.gotoStepByName("loopStarter") } 5) reset count Unnecessary step but I like getting into the habbit of clearing all properties after I'm done, and this groovy script will do it for you def loopProperties = testRunner.testCase.getTestStepByName("loopProperties") String[] removals = loopProperties.getPropertyNames() for(i=0;i<removals.size();i++){ loopProperties.removeProperty(removals[i]) } I've been thinking of doing something similar to this for the Excel DataSource scriptOlga_Tmentioned, well similar to the loopStarter step anyway - doing it completely through groovy so SoapUI free users can do it too, I just need to look into parsing exel files with groovy. Anyways I hope this helps you all, Mo0likes0Comments
