How to add test execution result with statuses of steps from API
Hello, i am working to integrate my c# and MSTest solution to cucumber studio, we are using BDD format scenarios/tests and need to add test execution results with status for each step, but as you see from the screenshot, there is no such support for it... can you help me with that?
Integrate Zephyr with Gitlab CI and Webdriver.io
We want to automate the creation of test cycles and marking the tests as executed. We have Zephyr for Jira 5.6.2 and A.T.O.M. We have about 70 test cases added to the Zephyr. Automated tests are written in Webdriver.io (JS framework based on selenium). Automatic tests are run in gitlab CI after each deploy to the RC server. Can these tools be integrated with each other? Flow: 1. Before each run, create a test cycle with all automated tests. (Based on template, label, component or something) 2. If the each test case passes in CI it is marked in jira as passed or if test case fail is marked as failed? 3. When all the tests are done report is made in zephyr? Is such a thing possible? Or should it be approached differently?
Database testing using soap ui/ready api ?
My REST web service populate db records as output. I can connect to db server using jdbc from soap ui. How can I compare data populated in REST request and JDBC request ? Below are few points to consider: 1. The column names in REST request and JDBC request are different but data is same. 2. The order of the records also differs. So how can I compare data output of REST service and JDBC request ? Is there any way I modify the output order in both REST service and JDBC request ?so that I canmake them to display in same order and compare the data. Also looking to automate the process for multiple requests. Please guide me with a right solution Hope I clearly explained the challenge. Looking for a right solution. Thanks, Ravi
oauth2 authorization dynamic test automation through event handlers
Hi, I created below script, hope you can benefit from it as well... I am relatively new to soapui & test automation, have less then 1 year experience with it and had no previous relevant coding background, so perhaps this might not be 100% efficient... but it does what it needs to do (for me :-) ). First of all, there are already a lot of features within soapui that allow you to dynamically use authorization. At project level you have the "Auth Manager" tab. You can create different profiles and specify for which operations to use which profile. The get access token screens did allow me to authenticate and retrieve an access token, but then we wanted to include my soapui testcases into the build pipeline (via TestRunner.bat command line execution) and there the integrated browser would not work. Even the automation of the browser windows (via front end automation getElementById etc.) would probably not work. So I went with the following approach which gave me sufficient customization possibilities while staying within a generic framework. It makes use of the soapui pro featureRequestFilter.filterRequest within "event handlers". First off, I need a test suite that will go get my authentication tokens. I have several, for easy demonstration I include 2, an oauth2 token (with access & refresh token validity for several hours) and an openid token generation, without refresh, but with limited 1 hour validity. I have 1 test suite which I will run at the start of the project, called "_Authorization". This includes 2 test cases, 1) "oauth2" and 2) "openid". For the first I have following test steps: 1.Groovy script to determine whether token is expired. 2. A rest request to refresh the token (if needed) called "REST Request Refresh_Token". This request uses the parameters client_id, grant_type (refresh), client_secret & refresh_token. (note: before being able to use this I had to manually obtain an access & refresh token first through grant_type (obtain authorization code). After this manual step I can get a new access token via the refresh token request automatically) 3. A datasink to write the new token (access & refresh) to an excel, together with the new expiration date etc. (if needed) 4. A datasource test step. This is were the event handler will go get my oauth2 token to inject in all my other test cases. The second test case "openid" will use a few more steps: 1. DataSource where I have an excel with all my specific "numbers" for which I want to generate a specific openid token. I can provide also a claim via this excel (first name, last name also, might depend on your openid client). In contrary to the above oauth2; the authorization code will be specific for certain parameterized request (for instance request for specificclients, id's, organisation numbers, ...). These request will be authorized with different authentication profiles (claims) that determine who can see what. 2. A rest request test step called "token" to obtain the tokens. This request uses the parameters grant_type, code & redirect_uri. 3. A groovy test step to get and set the properties for each number from my datasource. So here the tokens used by my test cases would be retrieved from the properties within the soapui project. 4. A datasink to write the new token to an excel, together with the new expiration date etc. I write it to an excel file as it is easier to have an overview via an external file. 5. A datasource loop that points back to 2. See below the scripts for: oauth2 test step 1: import java.text.DateFormat; import java.text.SimpleDateFormat; import java.util.Date; import groovy.time.TimeCategory TimeZone.setDefault(TimeZone.getTimeZone('CET')) testRunner.runTestStepByName("DataSource_external file") def expirydate = context.expand( '${DataSource_external file#expiry_date}' ) Date now = new Date() //expiry date is in string format. We need to convert to Date DateFormat datumformaat = new SimpleDateFormat("EEE MMM d HH:mm:ss z yyyy", Locale.US) Date expirydate_convertedtodate = datumformaat.parse(expirydate) if (expirydate_convertedtodate > now) { log.info "Access Token not expired. We run only the last step called DataSource_external file. This way all requests using this oauth2 authorization will be able to insert this token."; testRunner.gotoStepByName("DataSource_external file") } else { testRunner.runTestStepByName("DataSource_external file") //log.info "Access Token expired! Run the entire test suite to retrieve a new access & refresh token. If this step would fail we need to run manually an obtain access & refresh token rest request." def refresh_token = context.expand( '${DataSource_external file#refresh_token}' ) def client_secret = context.expand( '${DataSource_external file#client_secret}' ) def old_access_token = context.expand( '${DataSource_external file#access_token}' ) // we store it at test case level so rest request step to get the refresh token can use it. testRunner.testCase.setPropertyValue("refresh_token", refresh_token); testRunner.testCase.setPropertyValue("client_secret", client_secret); currentDate = new Date() use( TimeCategory ) { expirydate_minus1hour = currentDate - 3600.seconds expirydate_plus16hours = currentDate + 57600.seconds } testRunner.testCase.setPropertyValue("expirydate", expirydate_plus16hours.toString()) //I included an if statement here as I use also an old access token to test I do not get authorized. Old means that it was refreshed more than 1 hour ago, otherwise it would still be valid if (expirydate_convertedtodate < expirydate_minus1hour) { log.info ("The old access token expired on "+expirydate_convertedtodate+", this is more than 1 hour ago ("+expirydate_minus1hour+"). So this one is no longer valid and we can write it to property old_access_token") testRunner.testCase.setPropertyValue("old_access_token_DOSIS", old_access_token); } else { log.info ("Old access token expired on "+expirydate_convertedtodate+", this is LESS than 1 hour ago ("+expirydate_minus1hour+"). So this one would actually be still valid, so we do NOT write it to property old_access_token") } } Groovy script OpenId step 3: import groovy.time.TimeCategory TimeZone.setDefault(TimeZone.getTimeZone('CET')) def expires_in = context.expand( '${#Project#expires_in}' ) currentDate = new Date() use( TimeCategory ) { expirydate_plus1hour = currentDate + 3600.seconds } def numberA= context.expand( '${DataSource_OpenId#numberA}' ) def firstname = context.expand( '${DataSource_OpenId#first name}' ) def lastname = context.expand( '${DataSource_OpenId#last name}' ) def claim = context.expand( '${DataSource_OpenId#claim}' ) def description= context.expand( '${DataSource_OpenId#description}' ) def token_response = context.expand( '${Token#Response#$[\'id_token\']}' ) log.info ("For property "+omschrijving+" with numberA \""+numberA+"\" we got following token (expires on: "+ expirydate_plus1hour + "): " +token_response) testRunner.testCase.testSuite.project.setPropertyValue ("authorization_"+description, token_response) return expirydate_plus1hour And now the big kicker. The RequestFilter.filterRequest piece of code: //event creates custom header for ALL requests (overwriting the Authorization header if it was already provided in the test steps). import java.util.ArrayList; import java.util.List; import java.util.regex.*; def headers = request.requestHeaders // test step variabelen ophalen def requestname = request.getName() def numberA = context.getProperty("numberA") def numberB = context.getProperty("numberB") // based upon the operation we use in the test steps we will inject another authorization header. So the event listens to the compiled request and will insert the authorization header BEFORE it submits it. String operation = request.getOperation() // first operation is the actual get access & refresh tokens from the authorization service (oauth2 and openid). There is no authorization header needed for these operations. We use test suite _Authorization to dynamically call the authoriztion service and // write the results to an external excel file if ( operation == "RestResource: /authorization/oauth2/token"|| operation == "RestResource: /authorization/openid/token" ) { headers.remove("Authorization"); request.requestHeaders = headers; log.info ("teststep "+ requestname + " concerns operation " + operation + " . This is an authentication request oauth2 or openid, so we do not need to insert an authorization header.") } else // GENERIC - string "(403)", "(401)" or "(_negative_)" in de test step name - we do not pass authorization! This way we can test the actual authorzation, to see we get a 401 or 403 http response code if ( requestname.contains("(403)")|| requestname.contains("(401)")|| requestname.contains("(_negative_)") ) { headers.remove("Authorization"); request.requestHeaders = headers; log.info ("teststep "+ requestname + " authenticion testcase. So we do not insert Authorization heade (capital A). It is possible to manually insert an authorization header (no capital A, so we can distinguish between automatical insertion or manual)") } else // depending on the rest resource endpoint operations, I can insert specific authorization headers. For instance, below operations do not have an input parameter. To be able to generically determine which openid token to insert for each rest request test step // we include in the test step name square brackets with between it the number for which we want to get the openid token, for instance [12345 ] will go and insert the authorziation header token for number 12345, generated in the _Authentication test suite. if ( operation == "RestResource: /api/v1/abc/{id}"|| operation == "RestResource: /api/v1/def" ) { if ( requestnaam.contains("[") ) { Matcher openid = Pattern.compile("(?<=\\[).+?(?=\\])").matcher(requestnaam) def token = context.expand( '${#Project#authorization_'+openid[0]+'}' ) // log.info ("name of the test step is: " + requestnaam +" - with operation: " +operation) // log.info ("In the requstname we have \'[" +openid[0] + "]\', so we inject the authorization OpenId header like geneated in the _Authorization test suite: " +token) headers.remove("Authorization"); headers.put("Authorization", "bearer "+token); request.requestHeaders = headers; } else //Attention, in case of no "[" in the request name we do a default openid authorization { def token = context.expand( '${#Project#authorization_default}' ) log.info ("name of teststep is: " + requestnaam +" - with operation: " +operation) log.info (Default openid authorization, \${#Project#authorization_default} die resulteert in deze OpenId Authorization header: " +token) headers.remove("Authorization"); headers.put("Authorization", "bearer "+token); request.requestHeaders = headers; } } else // In below operation we will have an input parameter for the request. We will read that one and automatically inject the matching openid authorization header we got from the _Authorization test suite. if ( operation == "RestResource: /api/getdata/numberA/{numberA}"|| operation == "RestResource: /api/getotherdata/numberA/{numberA}" ) { def token = ""; if (numberA != null) { token = context.expand( '${#Project#OpenIdToken_'+numberA+'_soapui_default}' ) log.info ("Name of the teststep is: " + requestname +" - with operation: " +operation) log.info ("From the request we get parameter number " +numberA + " which results in following openid token: " +token) headers.remove("Authorization"); headers.put("Authorization", "bearer "+token); request.requestHeaders = headers; } else { log.info "No numberA found in the request parameter, so the result would probably be a bad request (400)"; throw new Exception("No numberA found in the request parameter, so the result would probably be a bad request (400)"); } } else // DEFAULT - we insert another oauth authorization header (not openid) valid for most of my operations { def authorization_oauth = context.expand('${#[_Authentication#_Authentication#DataSource_external file]#authorization_REST}'); headers.remove("Authorization"); headers.put("Authorization",authorization_oauth); request.requestHeaders = headers; log.info ("Name of test step is: " + requestname +" - with operation: " +operation) log.info ("Default event handler applied: Trigger event Authorization header with the general oauth2 token, retrieved from external file: "+authorization_oauth) }How attach files after an automated execution
A few weeks ago we purchased Zephyr Scale for Jira Cloud. I know how it works on Server version and now I implemented the cloud version for execution test, there I don't have any issues, however after an execution I can't upload files like the server version, where the url is like this "/testresult/{testResultId}/attachments" to post a file from a previous result. I'm trying to figure out how I can upload files, but checking the API information I could use "/automations/executions/custom" and I'm getting the following response: { "errorCode": 400, "message": "Couldn't find any mapped test cases", "status": "Bad Request" } How I can resolve this? Are there any alternatives for cloud version to accomplish this? In this place for Server version I was able to upload files after an execution with"/testresult/{testResultId}/attachments", for example:Reuse groovy scripts with a script library - Use case with dynamic environment url
Bundle all the frequently used scripts/ functions into a script library and use it where ever we want without duplicating the code. USE CASE: below piece of code is used in various test steps througout the project. But what if the environment url's change, then I would need to go and find all groovy test steps where these urls have changed and manually update them from "http://testwebsites.net/api/v1/agents/" to "http://newtestwebsites.net/api/v1/agents/" (= Use Case 1). And what if I want to reuse the same code snippet but not for "agents" but for "organisations"? (= Use Case 2) This piece of code below we want to isolate to be able to change it in one place only: def env = testRunner.testCase.testSuite.project.activeEnvironment.name def url switch (env) { case "TEST": url = "http://testwebsites.net/api/v1/agents/" break case "DEV": url = "http://devwebsites.net/api/v1/agents/" break; case "BETA": url = "http://betawebsites.net/api/v1/agents/" break; } Solution: See also: https://www.soapui.org/scripting---properties/scripting-and-the-script-library.html First we create a file Urls.groovy, put it in the scriptlibrary* , to isolate the url = Use Case 1: import com.eviware.soapui.model.testsuite.TestRunner //update: import com.eviware.soapui.impl.wsdl.panels.support.MockTestRunner--> this is only for running a groovy script test step individually, so use above TestRunner instead public class Urls { // Properties def testRunner; // Constructor Urls(TestRunner testRunner) { this.testRunner = testRunner; } // Methods def GetEnvUrl() { def env = testRunner.testCase.testSuite.project.activeEnvironment.name def url switch (env) { case "TEST": url = "http://testwebsites.net/api/v1/" break case "DEV": url = "http://devwebsites.net/api/v1/" break; case "BETA": url = "http://betawebsites.net/api/v1/" break; } return url; } } Notes: The reason why we need to use properties & constructor, is because we reuse some soapui built in objects, namely the testRunner. See the api pro documentation: http://www.soapui.org/apidocs/com/eviware/soapui/model/testsuite/TestRunner.html The method, which we named "GetEnvUrl()" contains the actual copy paste of the groovy script. The reason why we need to "return url" at the end is because url will only be known within this isolated piece of code and when would just call upon it within the groovy test step (without return in the class) it is not "transfered". A similar class for use case 2, namely AgentUrls.groovy: -----------------------------------------------------------------------------------------------------—------------------------------------------------------------------------------------— import com.eviware.soapui.model.testsuite.TestRunner class AgentUrls { def baseUrl = "agents/" // Properties def testRunner; // Constructor AgentUrls(TestRunner testRunner) { this.testRunner = testRunner; } // Methods String GetAllUrl() { def url = new Urls(testRunner).GetEnvUrl() + baseUrl return url; } } We can easily duplicate this class AgentUrls.groovy to make a OrganisationUrls.groovy by changing baseUrl to "organisations/" (or others). To call this piece of code again in the groovy test step, we can replace by this simple line: def url = new AgentUrls(testRunner).GetAllUrl() if you just need to have the url and not the "agents/" behind it: def url = new Urls(testRunner).GetEnvUrl() Note: the scriptlibrary can be set via an absolute path throught the settings/ReadyApi option (default = C:\Program Files\SmartBear\ReadyAPI-2.3.0\bin\scripts). If you share your codebase, in case of multiple testers or for running it in CIT build pipeline, we need a relative path. We can define it via Project Properties — SWcript Library option. We can do this the same way we define the relative path for the saving of our composite project:, also at project properties level (Resource Root) with for instance ${projectDir}. The result should always be visible in the readyApi log , for instance: Fri Nov 17 13:18:08 CET 2017:INFO:XXXXX\Script Library\Urls.groovy is new or has changed, reloading... Fri Nov 17 13:18:08 CET 2017:INFO:Resetting groovy class cache due to 1 modified file
