at062084
New Contributor
Joined 6 years ago
User Profile
User Widgets
Contributions
Re: TestEngine + Jenkins Plugin: Execute list of Project/Suite/Case SoapUI tests
Are there any plans to improve the usability of testrunner.sh towards executing selected testcases from many different SoapUI project files ? Or should the strategy be to extract and merge the testcases selected for automation into a SoapUI project dedicated to execution by testrunner.sh ? Thank you1.8KViews0likes0CommentsTestEngine + Jenkins Plugin: Execute list of Project/Suite/Case SoapUI tests
In their current implementation 1.) the TestEngine needs to be provided a zip with all required files by the calling client 2.) the TestEngine Jenkins plugin will execute one Project/Suite/Case per Jenkins job Problem: This setup does not scale to execute many Project/Suite/Case SoapUI tests in a row Suggestion: ad 1.) a.) Provide TestEngine the ability to access/checkout GIT repos containing SoapUI Projects b.) Add an API to TestEngine that will take the Project/Suite/Case/... parameters and pick all required files from above local checkout area --> Like this no files need to be created/transfered to the TestEngine server What still needs to be done is the specification of the list of Project/Suite/Case to be executed. This could be provided by one or more config files maintained in the GIT repo(s) ad 2.) In case the TestEngine runs on the Jenkins slave(s), the Jenkins job could - checkout the GIT repo(s) - locate the config file(s) containing the list of /Project/Suite/Case SoapUI Tests in the checkout area (e.g. via parameter) - loop the config file(s) calling the TestEngine API with Project/Suite/Case/... parameters --> like this - The TestEngine is running as a server (contrary to testruner.sh) - The invocation of many Project/Suite/Case's will run with minimal overhead (presumably msecs instead of secs) - all required files are taken from a repo -> any updates propagate to TestEngine automatically Remark: This type of automated execution of SoapUI Test from different Project/Suite/Case needs to complemented with a suitable test report (e.g. csv file with Project/Suite/Case/Step, timeStamp, Status, ExecTime, ErrorMsg, ...Re: readyapi 3.0 testrunner.sh: Startup sequence
From what i understand: - The TestEngine curl client needs to have the soapui project file plus any required files packed into a zip, that is created by export from the gui --> we've got quit a few projects, so rerunning the export on each update is not really an option - The TestEngine jenkins plugin allows to specify one project, testsuite and testcase --> we need to run TestCases from several/many projects, so not really an option either What would be more usefull is to have the TestEngine server access the Git repo that holds all the soapui projects, check these out, and then wait for instructions (curl requests) which project/testsuite/testcase to execute. Our automation setup loops over a list of projects, testsuites and testcases (csv file), calls testrunner for each, creates JUnit style reports, parses the <TestSuite>-<TestCase>-<TestStep>.*.txt result files for status, response time and error message into a csv file, runs statistical analysis and publishes results to Confluence.2.3KViews0likes2CommentsRe: readyapi 3.0 testrunner.sh: Startup sequence
Presumably loading of the readyapi 3 load and security modules (not included in soapui pro) adds to startup time. -> So question is these modules can be disabled somehow -> preferred option would be a new feature to pass a list of Projects/Suites/Cases to testrunner.sh2.4KViews0likes4Commentsreadyapi 3.0 testrunner.sh: Startup sequence
I'm executing a single testcase from a SoapUI project with testrunner.sh using the -s and -c arguments. While the actual soap test request only takes a few ms to complete, testrunner.sh startup and shutdown times sum up to around 10 seconds (!) Any chance to bring overhead time down ? I'm calling testrunner in a loop to execute testcases from different projects. Remark: SoapUI-5.5 community edition testrunner.sh is quite a bit faster (around 3-5 secs) Here's the testrunner.sh log ================================ = = READY_API_HOME = /export/local/share/SmartBear/ReadyAPI-3.0.0 = ================================ 18:04:30,947 INFO [license] ReadyAPI 3.0.0 (build 4280) started - System properties: {OS=Linux, User name=jenkins} 18:04:31,170 INFO [DefaultSoapUICore] SoapUI settings were initialized from [/home/jenkins/soapui-settings.xml] 18:04:31,200 INFO [SoapUIExtensionClassLoader] "/export/local/share/SmartBear/ReadyAPI-3.0.0/bin/ext/postgresql-9.3-1102.jdbc41.jar" has been added to the classpath. 18:04:31,201 INFO [SoapUIExtensionClassLoader] "/export/local/share/SmartBear/ReadyAPI-3.0.0/bin/ext/sqljdbc42.jar" has been added to the classpath. 18:04:31,201 WARN [SoapUIExtensionClassLoader] The custom library "/home/jenkins" does not exist. 18:04:31,238 INFO [DefaultSoapUICore] 2 JDBC drivers have been registered from the directory bin/ext: [org.postgresql.Driver, com.microsoft.sqlserver.jdbc.SQLServerDriver]. 18:04:31,283 INFO [SoapUIHttpClient] Initializing KeyStore 18:04:31,820 INFO [DefaultSoapUICore] Adding listeners from [/export/local/share/SmartBear/ReadyAPI-3.0.0/bin/listeners/demo-listeners.xml] 18:04:31,835 WARN [SoapUIJasperReportManager] The report directory "/home/jenkins" does not exist. The default report directory will be used. 18:04:33,113 INFO [PluginManager] 0 plugins loaded in 29 ms 18:04:33,113 INFO [DefaultSoapUICore] All plugins loaded 18:04:33,306 DEBUG [LicenseManagerServiceImpl] Trying to initialize license for ReadyAPI Bundle 18:04:33,312 DEBUG [LicenseManagerServiceImpl] No license found for ReadyAPI Bundle 18:04:33,316 DEBUG [LicenseManagerServiceImpl] Trying to initialize license for SoapUI 18:04:33,320 DEBUG [LicenseManagerServiceImpl] License host for ReadyAPI Bundle: ***** 18:04:33,321 DEBUG [LicenseManagerServiceImpl] No license found for SoapUI 18:04:33,325 DEBUG [LicenseManagerServiceImpl] Trying to initialize license for SoapUI 18:04:33,332 DEBUG [LicenseManagerServiceImpl] License host for SoapUI: **** 18:04:33,340 DEBUG [LicenseManagerServiceImpl] License host for SoapUI: **** 18:04:33,351 DEBUG [LicenseManagerServiceImpl] Valid license file found: /home/jenkins/.soapui/soapui.key, last modified: 2019-12-03T12:59:38.000+0100 18:04:33,495 DEBUG [LicenseManagerServiceImpl] Trying to initialize license for LoadUI 18:04:33,499 DEBUG [LicenseManagerServiceImpl] No license found for LoadUI 18:04:33,503 DEBUG [LicenseManagerServiceImpl] Trying to initialize license for ServiceV 18:04:33,506 DEBUG [LicenseManagerServiceImpl] License host for LoadUI: **** 18:04:33,507 DEBUG [LicenseManagerServiceImpl] No license found for ServiceV 18:04:33,513 DEBUG [LicenseManagerServiceImpl] License host for ServiceV: ***** 18:04:34,287 INFO [WsdlProject] Loaded project from [file:/export/TestAutomation/JenkinsWorkSpace/soapUI-projects/Services/ServicePinging.xml] 18:04:34,427 WARN [WsdlProject] Failed to find the "Default" environment. The default environment will be used. 18:04:34,460 INFO [SoapUIProTestCaseRunner] Setting project property [timestamp] to [date+%F%T] 18:04:34,463 INFO [SoapUIProTestCaseRunner] Running tests in the project [ServicePinging] 18:04:34,526 INFO [SoapUIProTestCaseRunner] Running TestCase [Contract_BusinessMgmtService] 18:04:34,531 INFO [SoapUIProTestCaseRunner] Running TestCase [Contract_BusinessMgmtService] 18:04:34,541 INFO [SoapUIProTestCaseRunner] Finished running TestCase [Contract_BusinessMgmtService], time taken: 8ms, status: PASS 18:04:34,544 INFO [SoapUIProTestCaseRunner] TestCase [BusinessMgmtService] finished with status [PASS] in 8ms 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:36,429 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:37,395 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:37,396 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:37,396 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:37,396 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:37,396 WARN [JRConditionalStyleFillerFactory] Conditional style should not have an 'isDefault' attribute. 18:04:37,396 WARN [JRConditionalStyleFillerFactory] Conditional style cannot have a 'style' attribute. 18:04:37,730 INFO [SoapUIProTestCaseRunner] Created report [TestCase Report] to path [/export/TestAutomation/SoapUIBuilds/entw/develop/Ping/000/PINGServices/ContractBusinessMgmtService/TestCase Report.html] 18:04:37,730 INFO [SoapUIProTestCaseRunner] Created report [TestCase Report] to path [/export/TestAutomation/SoapUIBuilds/entw/develop/Ping/000/PINGServices/ContractBusinessMgmtService/TestCase Report.csv] 18:04:37,731 INFO [license] ReadyAPI 3.0.0 shutting downSolved2.5KViews0likes6Comments