JsonPath Match and JsonPath RegEx Match assertions fail with SoapUI 5.7.0 (and worked with v5.6.1)
Hi, I'm totally new to SoapUI and I'm taking over the integration tests of a colleague who left the company.
Our security department requires us to use log4j 2.17 which is embedded in SoapUI 5.7.0. This requires us to upgrade our current version of SoapUI v5.6.1.
I tested few users scripts with SoapUI 5.7.0 and all run smoothly but the few integration tests fail. I've been searching for days without finding the solution to this issue. The issue is not related to our code since command testrunner.sh fails with the exact same parameters (see below).
I first run the mockservicerunner:
$ mockservicerunner.sh -m"REST MockService" -p 58475 test/integration/monitoring/soapui/soapui-project.xml
================================
=
= SOAPUI_HOME = /home/ubuntu/SmartBear/SoapUI-5.7.0
=
================================
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
SoapUI 5.7.0 MockService Runner
17:40:40,385 INFO [DefaultSoapUICore] Creating new settings at [/home/ubuntu/soapui-settings.xml]
17:40:45,326 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/ready-mqtt-plugin-dist.jar]
17:40:45,334 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/readyapi-postman-1.0.1.jar]
17:40:46,571 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/ready-uxm-plugin-os-1.0.3.jar]
17:40:46,991 INFO [PluginManager] 3 plugins loaded in 1661 ms
17:40:46,992 INFO [DefaultSoapUICore] All plugins loaded
17:40:50,191 INFO [WsdlProject] Loaded project from [file:/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/test/integration/monitoring/soapui/soapui-project.xml]
17:40:50,201 INFO [SoapUIMockServiceRunner] Running MockService [REST MockService] in project [Hestia Integration]
17:40:50,201 INFO [SoapUIMockServiceRunner] Press any key to terminate
17:40:50,797 INFO [JettyMockEngine] Started mockService [REST MockService] on port [58475] at path [/]
17:40:50,803 INFO [SoapUIMockServiceRunner] MockService started on port 58475 at path [/]
17:40:50,803 INFO [SoapUIMockServiceRunner] Started 1 runner
Press any key to terminate...
17:43:09,496 INFO [log] Path: /pet/findByStatus
17:43:09,623 INFO [log] QueryString: status=available
17:43:09,707 INFO [SoapUIMockServiceRunner] Handled request 1; [/pet/findByStatus] with [Available] in [0ms] at [2022-06-28 17:43:09.523]
17:43:10,732 INFO [log] Path: /pet/findByStatus
17:43:10,733 INFO [log] QueryString: status=available
17:43:10,749 INFO [SoapUIMockServiceRunner] Handled request 2; [/pet/findByStatus] with [Available] in [0ms] at [2022-06-28 17:43:10.733]
17:43:10,770 INFO [log] Path: /pet/findByStatus
17:43:10,771 INFO [log] QueryString: status=available
17:43:10,790 INFO [SoapUIMockServiceRunner] Handled request 3; [/pet/findByStatus] with [Available] in [0ms] at [2022-06-28 17:43:10.770]
17:43:10,914 INFO [log] Path: /pet/findByStatus
17:43:10,917 INFO [log] QueryString: status=available
17:43:10,931 INFO [SoapUIMockServiceRunner] Handled request 4; [/pet/findByStatus] with [Available] in [0ms] at [2022-06-28 17:43:10.915]
17:43:10,950 INFO [log] Path: /pet/findByStatus
17:43:10,951 INFO [log] QueryString: status=available
17:43:10,963 INFO [SoapUIMockServiceRunner] Handled request 5; [/pet/findByStatus] with [Available] in [0ms] at [2022-06-28 17:43:10.951]
Then I run the test with testrunner:
$ testrunner.sh -rAI -Duser.timezone=UTC -Dlogfilename=TestREST_-_test_simple_success_latest.log -Derrfilename=TestREST_-_test_simple_success_errors.log -Dsoapui.log.dir=test/integration/monitoring/soapui/rest/simple/success/logs -s 'Simple - REST Only - success' -h localhost:58475 -e http://localhost:58475 test/integration/monitoring/soapui/rest/simple/success/project.xml
================================
=
= SOAPUI_HOME = /home/ubuntu/SmartBear/SoapUI-5.7.0
=
================================
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
SoapUI 5.7.0 TestCase Runner
17:59:39,500 INFO [DefaultSoapUICore] Creating new settings at [/home/ubuntu/soapui-settings.xml]
17:59:40,439 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/ready-mqtt-plugin-dist.jar]
17:59:40,450 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/readyapi-postman-1.0.1.jar]
17:59:41,366 INFO [PluginManager] Adding plugin from [/home/ubuntu/.soapuios/plugins/ready-uxm-plugin-os-1.0.3.jar]
17:59:41,810 INFO [PluginManager] 3 plugins loaded in 1374 ms
17:59:41,810 INFO [DefaultSoapUICore] All plugins loaded
17:59:43,215 INFO [WsdlProject] Loaded project from [file:/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/test/integration/monitoring/soapui/rest/simple/success/project.xml]
17:59:43,219 INFO [SoapUITestCaseRunner] Running SoapUI tests in project [Hestia Integration]
17:59:43,220 INFO [SoapUITestCaseRunner] Running TestSuite [Simple - REST Only - success], runType = SEQUENTIAL
17:59:43,226 INFO [SoapUITestCaseRunner] Running SoapUI testcase [No assertions]
17:59:43,230 INFO [SoapUITestCaseRunner] running step [Get available pets]
17:59:44,245 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [No assertions], time taken: 731ms, status: FINISHED
17:59:44,254 INFO [SoapUITestCaseRunner] Running SoapUI testcase [Status code assertion]
17:59:44,254 INFO [SoapUITestCaseRunner] running step [Get available pets]
17:59:44,285 INFO [SoapUITestCaseRunner] Assertion [Valid HTTP Status Codes] has status VALID
17:59:44,290 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [Status code assertion], time taken: 12ms, status: FINISHED
17:59:44,290 INFO [SoapUITestCaseRunner] Running SoapUI testcase [JSON path existence assertion]
17:59:44,291 INFO [SoapUITestCaseRunner] running step [Get available pets]
17:59:44,346 INFO [SoapUITestCaseRunner] Assertion [JsonPath Existence Match - Id is returned] has status VALID
17:59:44,347 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [JSON path existence assertion], time taken: 2ms, status: FINISHED
17:59:44,350 INFO [SoapUITestCaseRunner] Running SoapUI testcase [JSON path match assertion]
17:59:44,350 INFO [SoapUITestCaseRunner] running step [Get available pets]
17:59:44,372 INFO [SoapUITestCaseRunner] Assertion [JsonPath Match - First status is available] has status FAILED
17:59:44,372 ERROR [SoapUITestCaseRunner] ASSERTION FAILED -> Comparison failed for path [$..status[0]], expecting [available], actual was [[]]
17:59:44,372 ERROR [SoapUITestCaseRunner] ASSERTION FAILED -> Comparison failed for path [$..status[0]], expecting [available], actual was [[]]
17:59:44,373 ERROR [SoapUITestCaseRunner] Get available pets failed, exporting to [/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/Simple__REST_Only__success/JSON_path_match_assertion/Get_available_pets-0-FAILED.txt]
17:59:44,373 ERROR [SoapUITestCaseRunner] Get available pets failed, exporting to [/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/Simple__REST_Only__success/JSON_path_match_assertion/Get_available_pets-0-FAILED.txt]
17:59:44,373 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [JSON path match assertion], time taken: 3ms, status: FAILED
17:59:44,374 INFO [SoapUITestCaseRunner] Running SoapUI testcase [JSON path regex match assertion]
17:59:44,374 INFO [SoapUITestCaseRunner] running step [Get available pets]
17:59:44,399 INFO [SoapUITestCaseRunner] Assertion [JsonPath RegEx Match - Pet name is valid] has status FAILED
17:59:44,400 ERROR [SoapUITestCaseRunner] ASSERTION FAILED -> Comparison failed for path [$..name[0]], expecting [true], actual was [false]
17:59:44,400 ERROR [SoapUITestCaseRunner] ASSERTION FAILED -> Comparison failed for path [$..name[0]], expecting [true], actual was [false]
17:59:44,400 ERROR [SoapUITestCaseRunner] Get available pets failed, exporting to [/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/Simple__REST_Only__success/JSON_path_regex_match_assertion/Get_available_pets-0-FAILED.txt]
17:59:44,400 ERROR [SoapUITestCaseRunner] Get available pets failed, exporting to [/home/ubuntu/MyRepo/git/TSM/hestia-software-scripts/Simple__REST_Only__success/JSON_path_regex_match_assertion/Get_available_pets-0-FAILED.txt]
17:59:44,402 INFO [SoapUITestCaseRunner] Finished running SoapUI testcase [JSON path regex match assertion], time taken: 2ms, status: FAILED
17:59:44,405 INFO [SoapUITestCaseRunner] TestSuite [Simple - REST Only - success] finished with status [FAILED] in 1179ms
SoapUI 5.7.0 TestCaseRunner Summary
-----------------------------
Time Taken: 1185ms
Total TestSuites: 1
Total TestCases: 5 (2 failed)
Total TestSteps: 5
Total Request Assertions: 4
Total Failed Assertions: 2
Total Exported Results: 5
I know that there's a bug with "JsonPath Match" assertions in SoapUI 6.7.0 but replacing them with " XPath Match" does not solve the pb.
Find attached the entry files.
Any help would be greatly appreciated.
Best regards,
Sabrina
I finally figured out what the pb was: this is most likely related to the upgrade of the Jayway jsonpath library from 0.9.1 (SOAPUI 5.6) to 2.4.0 (SOAPUI 5.7). See SOAPUI release notes:
We've updated a number of third-party libraries, including the following:
- Apache Log4j library to 2.17.1.
- Jayway jsonpath to 2.4.0.
In the project file, replacing "$..status[0]" with " $[0].status" solved the issue.
I used this website for JSONPath implementation differences and validation: https://jsonpath.herokuapp.com/.