Test Runner Error

Question

We use Jenkins to execute our tests automatically using the test runner. We execute them using the following command:

CALL "%SOAPUI_HOME%\bin\testrunner.bat" %%g -PEnvironment=%RUN_ENVIRONMENT% -Ijf TestResults\%%~ng

So to test this I've been running the following, just like jenkins is:

testrunner.bat "C:\....\Test.xml" -PEnvironment={ENV} -Ijf TestResults\Test

When I run this on my development machine, everything works just fine and I see the following output:

OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=permit; support was removed in 17.0
SoapUI 5.8.0 TestCase Runner
Configuring log4j from [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\soapui-log4j.xml]
11:35:24,633 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\Configuration.jar] to extensions classpath
11:35:24,649 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\ConfigurationUI.jar] to extensions classpath
11:35:24,649 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\Library.jar] to extensions classpath
11:35:24,649 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\ProjectRunner.jar] to extensions classpath
11:35:24,649 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\sqljdbc4.jar] to extensions classpath
11:35:24,903 INFO [DefaultSoapUICore] initialized soapui-settings from [C:\Users\...\soapui-settings.xml]
11:35:32,908 INFO [DefaultSoapUICore] Adding listeners from [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\listeners\AutomationListener-listeners.xml]
11:35:33,604 INFO [DefaultSoapUICore] Adding listener [class AutomationListener]
11:35:35,207 INFO [PluginManager] Adding plugin from [C:\Users\...\.soapuios\plugins\ready-mqtt-plugin-dist.jar]
11:35:35,207 INFO [PluginManager] Adding plugin from [C:\Users\...\.soapuios\plugins\readyapi-postman-1.0.1.jar]
11:35:36,813 INFO [PluginManager] Adding plugin from [C:\Users\...\.soapuios\plugins\ready-uxm-plugin-os-1.0.3.jar]
11:35:37,610 INFO [PluginManager] 3 plugins loaded in 2428 ms
11:35:37,610 INFO [DefaultSoapUICore] All plugins loaded
11:35:39,694 INFO [WsdlProject] Loaded project from [file:/C:/..../Test.xml]
11:35:39,705 INFO [SoapUITestCaseRunner] Setting project property [Environment] to [ENV]
11:35:39,705 INFO [SoapUITestCaseRunner] Running SoapUI tests in project [Test]
11:35:39,705 INFO [SoapUITestCaseRunner] Running Project [CodeTypes], runType = SEQUENTIAL
11:35:40,937 INFO [SoapUITestCaseRunner] Running SoapUI testcase [Returns]
11:35:40,953 INFO [SoapUITestCaseRunner] running step [...]
11:35:49,950 INFO [SoapUITestCaseRunner] Assertion [...] has status VALID
11:35:49,951 INFO [SoapUITestCaseRunner] Assertion [...] has status VALID
11:35:49,953 INFO [SoapUITestCaseRunner] Assertion [...] has status VALID
11:35:49,955 INFO [SoapUITestCaseRunner] Assertion [...] has status VALID
11:35:49,955 INFO [SoapUITestCaseRunner] Assertion [...] has status VALID
11:35:49,969 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [...], time taken: 6452ms, status: FINISHED
11:35:49,985 INFO [SoapUITestCaseRunner] Project [CodeTypes] finished with status [FINISHED] in 10272ms

This is what I expect. However, when I run it on the jenkins server I get the following:

C:\Program Files\SmartBear\SoapUI-5.8.0\bin>testrunner.bat "{TESTPATH}\Test.xml" -PEnvironment={ENV} -Ijf TestResults\Test
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=permit; support was removed in 17.0
SoapUI 5.8.0 TestCase Runner
Configuring log4j from [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\soapui-log4j.xml]
12:25:40,839 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\Configuration.jar] to extensions classpath
12:25:40,848 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\ConfigurationUI.jar] to extensions classpath
12:25:40,848 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\Library.jar] to extensions classpath
12:25:40,849 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\ProjectRunner.jar] to extensions classpath
12:25:40,849 INFO [SoapUI] Adding [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\ext\sqljdbc4.jar] to extensions classpath
12:25:41,205 INFO [DefaultSoapUICore] initialized soapui-settings from [C:\Users\....\soapui-settings.xml]
12:25:44,172 INFO [DefaultSoapUICore] Adding listeners from [C:\Program Files\SmartBear\SoapUI-5.8.0\bin\listeners\AutomationListener-listeners.xml]
12:25:44,857 INFO [DefaultSoapUICore] Adding listener [class AutomationListener]
12:25:46,649 INFO [PluginManager] Adding plugin from [C:\Users\....\.soapuios\plugins\ready-mqtt-plugin-dist.jar]
12:25:46,693 INFO [PluginManager] Adding plugin from [C:\Users\....\.soapuios\plugins\readyapi-postman-1.0.1.jar]
12:25:48,862 INFO [PluginManager] Adding plugin from [C:\Users\....\.soapuios\plugins\ready-uxm-plugin-os-1.0.3.jar]
12:25:49,945 INFO [PluginManager] 3 plugins loaded in 3298 ms
12:25:49,953 INFO [DefaultSoapUICore] All plugins loaded
12:25:52,250 INFO [WsdlProject] Loaded project from [file:/C:/..../Tests/Test.xml]
12:25:52,262 INFO [SoapUITestCaseRunner] Setting project property [Environment] to [ENV]
12:25:52,267 INFO [SoapUITestCaseRunner] Running SoapUI tests in project [TESTNAME]
12:25:52,269 INFO [SoapUITestCaseRunner] Running Project [TESTNAME], runType = SEQUENTIAL
12:25:54,115 ERROR [AbstractTestRunner] Exception during Test Execution
groovy.lang.MissingPropertyException: No such property: size for class: groovy.sql.GroovyRowResult
at groovy.sql.GroovyRowResult.getProperty(GroovyRowResult.java:55) ~[groovy-sql-3.0.19.jar:3.0.19]
at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:190) ~[groovy-3.0.19.jar:3.0.19]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAtIterable(DefaultGroovyMethods.java:8759) ~[groovy-3.0.19.jar:3.0.19]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAt(DefaultGroovyMethods.java:8751) ~[groovy-3.0.19.jar:3.0.19]

and the stack trace continues.

We do have a custom library that we wrote for data assertions and building/tearing down test data using sql scripts. And it runs just fine EXCEPT on the jenkins servers. There are 3 servers with this issue currently.

When I load the SoapUI GUI and run the tests, everything passes just fine. It's only when I execute the test from the command line it seems to have an issue.

I copied my entire install folder of SoapUI from my development machine to the Jenkins server, thinking maybe it was an issue with the install, and I have the same results.

At this point I'm out of ideas on why SoapUI is throwing this error when running a test through the command line on these specific servers.

All of them have the same JDK installed (23), the same version of groovy (3.0.19), the same version of SoapUI (5.8).

Any suggestions on other things I can look at would be greatly appreciated.

bigdaddyduergar · Answer

So we found what the issue is, but not the why, so this is our theory:

using rows.size works in the UI, as it allows an implicit function call to work. Something changed in a windows update (Updates rolled on the original vm with the issue in december, my local machine in January, which is when my local started having the same issue) that no longer allows implicit function calls and now requires explicit function calls. Something we're assuming changed with a windows or java update that was rolled out, that won't allow the test runner to make that decision of is it a function or a property.

The fix was just making sure .size was called using .size(). But why this stopped working after 10 years of being fine is still unknown.

Forum Discussion

Test Runner Error

1 Reply

Recent Discussions

How do I add a REST query string without the 'name='

Rest step assertions written in SoapUI 5.5.0 are failing in SoapUI 5.8.0

SoupUI 5.7.1

Related Content

UI (browser) testing via Gitlab runner

Test Runner bat via Octopus Deploy

Test error after rerun tests