Jenkins does not recognize ReadyApi maven plug-in test failures as unstable build

[chilber]

We're using Ready Api 2.3.0 with the ready-api-maven-plugin 2.3.0 to run our tests, which works fine.

Although some tests fail, Jenkins does not recognice the build as unstable, as it should when tests are failing.

ReadyAPI 2.3.0 TestCaseRunner Summary
-----------------------------
Time Taken: 16172ms
Total TestSuites: 0
Total TestCases: 2 (2 failed)
Total TestSteps: 686
Total Request Assertions: 5
Total Failed Assertions: 0
Total Exported Results: 2

19:40:57,286 INFO  [SoapUIProTestCaseRunner] Created report at file:/C:/vhvEntw/Jenkins/workspace/Testautomatisierung/SOAP_UI_Vertrieb/Vertrieb_SoapUI_TAA_Kfz_S/poms/kfz/s/target/TestOutput/index.html
[INFO] ReadyAPI 2.3.0 shutting down
[JENKINS] Zeichne Testergebnisse auf
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.889 s
[INFO] Finished at: 2019-07-25T19:40:57+02:00
[INFO] Final Memory: 78M/1189M
[INFO] ------------------------------------------------------------------------

 Here's the config that writes the results to the default maven location:

<configuration>
<projectFile>
../../../services/kfz/auto/TarifrechnerWS_KFZTarifrechnerService_v0118
</projectFile>
<projectProperties>
<value>taa_endpoint=${#Project#taa_Endpoint_S}</value>
<value>Verarbeitungsebene=S</value>
</projectProperties>
<testCase>Regression Tarif 01.10.18</testCase>
<junitReport>true</junitReport>
<testFailIgnore>true</testFailIgnore>
<printReport>true</printReport>
<reports>${basedir}/target/surefire-reports</reports>
<outputFolder>${basedir}/target/TestOutput</outputFolder>
</configuration>

content:

Does anyone have an idea why Jenkins does not consider this build as unstable?

This works when using the soapui-maven-plugin the same way...

Thanks

Carsten

[JustinM89 · Accepted solution]

You are ignoring test failures in your plugin config in this line:

<testFailIgnore>true</testFailIgnore>

Actually, that's not quite correct, according to the documentation, this property determines if a test will continue running if it encounters an error:

Continues the test run even if an assertion triggers.
Possible values: true, false.

After your test runs, you can use the JUnit plugin to parse the results, and that will pass/fail the build.

[Olga_T]

Thank you, @JustinM89!

@chilber, was this information helpful?

[chilber]

Thanks for the answer, JustinM89.

But this does not adress my problem. I don't want the build to fail if a test fails, I want the build to be marked as unstable. Please consider the Jenkins terminology here: https://wiki.jenkins.io/display/JENKINS/Terminology

Thanks

Carsten

[JustinM89]

I should clarify, the JUnit plugin will mark the build unstable (not failed) if there are any test failures.

[Olga_T]

Thanks for the clarification, @JustinM89.

@chilber, does it work for you in this case?