ReadyAPI-Azure : how to increase timeout when running testcases in testrunner in azure pipeline
I have been running some test cases in ReadyAPI windows application and increase the timeout for the testcases so they runs OK, but when running with the testrunner-commandline in Azure pipeline, some of the test cases times out. These test calls Selenide (Selenium) teststeps, where we have implemented some automated GUI test and they take a longtime to execute. Is there an options for the testrunner - test cases timeout, there is a testrunner bat file but does not include an obvious testcase-timeout option.Solved2.4KViews0likes7CommentsTestEngine + Jenkins Plugin: Execute list of Project/Suite/Case SoapUI tests
In their current implementation 1.) the TestEngine needs to be provided a zip with all required files by the calling client 2.) the TestEngine Jenkins plugin will execute one Project/Suite/Case per Jenkins job Problem: This setup does not scale to execute many Project/Suite/Case SoapUI tests in a row Suggestion: ad 1.) a.) Provide TestEngine the ability to access/checkout GIT repos containing SoapUI Projects b.) Add an API to TestEngine that will take the Project/Suite/Case/... parameters and pick all required files from above local checkout area --> Like this no files need to be created/transfered to the TestEngine server What still needs to be done is the specification of the list of Project/Suite/Case to be executed. This could be provided by one or more config files maintained in the GIT repo(s) ad 2.) In case the TestEngine runs on the Jenkins slave(s), the Jenkins job could - checkout the GIT repo(s) - locate the config file(s) containing the list of /Project/Suite/Case SoapUI Tests in the checkout area (e.g. via parameter) - loop the config file(s) calling the TestEngine API with Project/Suite/Case/... parameters --> like this - The TestEngine is running as a server (contrary to testruner.sh) - The invocation of many Project/Suite/Case's will run with minimal overhead (presumably msecs instead of secs) - all required files are taken from a repo -> any updates propagate to TestEngine automatically Remark: This type of automated execution of SoapUI Test from different Project/Suite/Case needs to complemented with a suitable test report (e.g. csv file with Project/Suite/Case/Step, timeStamp, Status, ExecTime, ErrorMsg, ...1.9KViews1like1CommentReadyAPI and TestEngine with Zephyr Scale
In this post, we are going to talk about using some of the SmartBear API testing tools and writing the results in an automated fashion to Zephyr Scale. But first let's take a step back and discuss a general approach of using testing tools with test management tools. At a manual level, when creating simple Multi-Protocol API tests, we might use a lighter weight tool like SoapUI Open Source, a great tool for quick and basic functional and performance API tests. After we trigger the test manually, we as humans would parse the report for any details on the test run, and to track it would update a spreadsheet, testcase in Jira, but somewhere. This sequence is important to note for when we start to advance to an automated workflow. Now let's talk about automating this process. We execute our API tests in an ephemeral environment, publish the results to a relative location, and then pass those results from that relative location to a test management platform. It is quite simple. In this workflow it tends to be easier to use a tool that has native integrations into CI platforms like Jenkins and Azure Dev Ops, that make managing our executions in these ephemeral environments much easier. Here, ReadyAPI, or TestEngine is the clear choice as they both seamlessly integrate into any CI-CD system and have native integrations into Jenkins, Azure, and many more. Now that sequence I mentioned earlier is very important. We will execute our API tests, and then pass the results into Zephyr Scale as step 2 in the pipeline. Here is a look at a scripted pipeline approach, note that there is freestyle job options available in Jenkins too. ReadyAPI Jenkins Plugin: https://support.smartbear.com/readyapi/docs/integrations/jenkins.html TestEngine Jenkins Plugin: https://support.smartbear.com/testengine/docs/admin/jenkins.html node { stage('Run API Tests') { // Run the API Tests using ReadyAPI or TestEngine stage('Pass Results') { //Pass Results to Zephyr Scale } } Now we need to be a little more specific with the 'Run API Tests' Stage. Above I mention when we run our API Tests in an automated fashion, we need to write the results to a relative location so that we can then send those results to Zephyr Scale from that location. Both ReadyAPI and TestEngine allow us to write results to locations as part of the command-line or native integrations. I will show command- line options for ReadyAPI and UI native integration for TestEngine but both options are available for both tools. Starting with ReadyAPI, testrunner CLI, the -f and -F flags represent the directory we are writing to, and the report format, respectively. ReadyAPI offers reports in PDF, XLS, HTML, RTF, CSV, TXT and XML, but the automation recommendation would be to pass results in the Junit-XML option. At a basic level we need this: testrunner.bat [optional-arguments] <test-project> And we need to specify -f and -F testrunner.bat -f<directory> -F<args> <test-project> and with -f requiring a relative directory, that can change based on the CI system. I will use Azure Dev Ops for both my examples here. In Azure I pull my test cases from the $(System.DefaultWorkingDirectory), which contains my git repo. In Azure I publish results to the $(Common.TestResultsDirectory) An example full command would look like this: "C:\Program Files\SmartBear\ReadyAPI-3.40.1\bin\testrunner.bat" -r -a -j -f$(Common.TestResultsDirectory) "-RJUnit-Style HTML Report" -FXML "-EDefault environment" "$(System.DefaultWorkingDirectory)/" With TestEngine it's very similar, but I am highlighting it through the native integration, note the publish test results and save Junit report option enabled below: Now lastly, we need to send the results to Zephyr Scale from the pipeline, before our release is over. I think it's easiest with the Zephyr Scale API: https://support.smartbear.com/zephyr-squad-cloud/docs/api/index.html along with the Auto-Create Test Case option to true. The command below is a basic example, and replica of the one seen in the Azure Pipeline screenshot. curl -H "Authorization: Bearer Zephyr-Scale-Token-Here" -F file= Relative-Location-of-Report-Here\report.xml;type=application/xml "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey=Project-Key-Here&autoCreateTestCases=true" After you modify the API token, Relative location, and project key you are good to run the pipeline. When we jump to Jira, we can see inside Zephyr Scale that the results are populating. Even with transactional Data on the failed test steps1.7KViews0likes0CommentsDifference result between TestEngine and ReadyAPI
Hi Hi build my test with ready API, my test is OK, my test contains a groovy scripts however When I execute this test in TestEngine this same test is Failed. I add in this parametre : By default, Groovy scripting has limitations on writing data to properties and files. If you need fully functional Groovy scripts in your tests, you can turn off these restrictions. Turning off the restrictions can cause security vulnerabilities. To turn off the limitations, you need to set the groovy.allow.all system property to true. The way you do this depends on how you run TestEngine: but in my Test on TestEngine I have this error on my grouvy script., why the behaviour is different between TestEngine and ReadyAPI IDE ? Error. java.net.UnknownHostException: https Error. error at line: 201.1KViews1like4CommentsLicense file installation in Docker
I am trying to install TestEngine with Dockerfile. Into a K8S Cluster. I have simply this in my Dockerfile. FROMsmartbear/readyapi-testengine:latest But, I am seeing error 16:20:41,864 ERROR [QuartzJobsScheduler] Missing License Details, scheduleLicenseExpirationJob would be skipped 16:20:41,864 ERROR [QuartzJobsScheduler] Missing License Details, scheduleLicenseTokenUpdateJob would be skipped All, I want to know is where to install the license file in the Dockerfile. I HAVE the license file .key file with me. not on the license server. I tried copying the file to the / in the Dockerfile. Like this: COPYabcd.key/abcd.key But that didn't help. So, please I am looking for the path where the license file need to be placeSolved1.1KViews0likes1CommentGetting results when using TestEngine as a container? ...
We use ReadyAPI stand-alone, and are going down the path of using TestEngine to run tests as part of our CI/CD pipeline. I see that TestEngine when running continuously has a web interface where I think you can see results. But how is this handled when using it as a container that instantiates when needed and then goes-away when the tests are complete? How do you see results? Thanks very much in advance.Solved1KViews0likes3CommentsReadyAPI TestEngine proxy bypass?
Hi all, With the recent release of TestEngine 1.26.0 it now supports using a proxy server. This is really helpful because I can now use the API to install (and uninstall) the license used by TestEngine. However it seems all the traffic from the tests being run in TestEngine also defaults to using the proxy - which is not so helpful for tests of internal systems that can not be accessed via the proxy. So what I'm looking for is a way to add exceptions to the proxy server? Since the the proxy is configured in the 'ReadyAPITestEngine.vmoptions' files I tried adding "-Dhttp.nonProxyHosts=mycompanys.domain.com". But doesn't seem to change TestEngine's behavior. Does anyone have any ideas of how to encourage TestEngine to not use the proxy when running tests (but still leave it there for adding and removing the license)? (in a similar way to just turning the proxy off from the button in ReadyAPI) Thanks EddieSolved999Views0likes1CommentHow to reuse a groovy teststep inside the project assertion scripts
I have a working groovy test script (a helper Cookie class with instance creation inside the script): context.setProperty("Cookie", new Cookie(log, context, testRunner)) and re-using the Cookie: project = testRunner.testCase.testSuite.project; testStep = project.testSuites["Helpers"].testCases["Objects"].testSteps["Cookie"]; testStep.run(testRunner, context); log.info context.Cookie.info() Runs fine from other groovy scripts in the project. But I need to re-use the code in the assertion scripts as well. Tried import com.eviware.soapui.support.types.StringToObjectMap import com.eviware.soapui.impl.wsdl.testcase.WsdlTestCaseRunner //def Runner = new WsdlTestCaseRunner(messageExchange.modelItem.testStep.testCase, new StringToObjectMap()); def Runner = new com.eviware.soapui.impl.wsdl.testcase.WsdlTestCaseRunner( messageExchange.modelItem.testStep.testCase, new com.eviware.soapui.support.types.StringToObjectMap()); def mySteps= messageExchange.modelItem.testStep.testCase.testSuite.project.testSuites["Helpers"].testCases["Objects"].testSteps["Cookie"]; mySteps.run(Runner,context); log.info mySteps.name log.info context.Cookie.info() but got name is ok; Runner is a valid instance. But Cookie is null. Error in assertion script of the [Login - Unregistered Mobile User Login] test step: Cannot invoke method info() on null object Can anyone help me solve the problem, please... Script library works fine but is not an appropriate case as it requires additional permissions to write to the destination folder - a jenkins job on a cluster...Solved979Views0likes1CommentLicense is not installed or has expired
We have users that are connecting to our ReadyAPI TestEngine server that get this this error every couple of days, Error, ReadyAPI TestEngine license is not installed or has expired. This is corrected each time by reclaiming the license, but it is holding up work until it is corrected. Our license does not expire until June. Has anyone seen experienced this and found a fix?Solved965Views0likes1CommentPermission issue in Test Engine
I am trying to run Ready api tests in Test Engine. C:\Program Files\SmartBear\ReadyAPITestEngine\bin>ReadyAPITestEngine.exe got started from Terminal successfullyon remote desktop. When I am running test jobs on Jenkins I am getting below error: 21:55:00,269 INFO [org.eclipse.jetty.server.RequestLog] 10.163.195.160 - - [01/Aug/2022:21:55:00 +0000] "POST /api/v1/testjobs HTTP/1.1" 400 75 "-" "Jersey/2.32 (HttpUrlConnection 11.0.14)" 94 21:55:00,442 INFO [com.smartbear.ready.testserver.server.ReadyApiTestServerSecurityPolicy] Permission denied: ("java.io.FilePermission" "C:\Users\prajvaidya\AppData\Local\Temp\ts_project_aa50ef98-067c-49b8-a9ef-bf730d7eb486\.git\refs\remotes" "write") I checked I have all the permission on remote desktop and to the above folder as well.933Views0likes3Comments