Forum Discussion

chilber's avatar
chilber
Occasional Contributor
6 years ago

Jenkins does not recognize ReadyApi maven plug-in test failures as unstable build

We're using Ready Api 2.3.0 with the ready-api-maven-plugin 2.3.0 to run our tests, which works fine.

Although some tests fail, Jenkins does not recognice the build as unstable, as it should when tests are failing.

 

ReadyAPI 2.3.0 TestCaseRunner Summary
-----------------------------
Time Taken: 16172ms
Total TestSuites: 0
Total TestCases: 2 (2 failed)
Total TestSteps: 686
Total Request Assertions: 5
Total Failed Assertions: 0
Total Exported Results: 2

19:40:57,286 INFO  [SoapUIProTestCaseRunner] Created report at file:/C:/vhvEntw/Jenkins/workspace/Testautomatisierung/SOAP_UI_Vertrieb/Vertrieb_SoapUI_TAA_Kfz_S/poms/kfz/s/target/TestOutput/index.html
[INFO] ReadyAPI 2.3.0 shutting down
[JENKINS] Zeichne Testergebnisse auf
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.889 s
[INFO] Finished at: 2019-07-25T19:40:57+02:00
[INFO] Final Memory: 78M/1189M
[INFO] ------------------------------------------------------------------------

 Here's the config that writes the results to the default maven location:

 

 

<configuration>
<projectFile>
../../../services/kfz/auto/TarifrechnerWS_KFZTarifrechnerService_v0118
</projectFile>
<projectProperties>
<value>taa_endpoint=${#Project#taa_Endpoint_S}</value>
<value>Verarbeitungsebene=S</value>
</projectProperties>
<testCase>Regression Tarif 01.10.18</testCase>
<junitReport>true</junitReport>
<testFailIgnore>true</testFailIgnore>
<printReport>true</printReport>
<reports>${basedir}/target/surefire-reports</reports>
<outputFolder>${basedir}/target/TestOutput</outputFolder>
</configuration>

content:

 

 

Does anyone have an idea why Jenkins does not consider this build as unstable?

This works when using the soapui-maven-plugin the same way...

 

Thanks

Carsten

  • JustinM89's avatar
    JustinM89
    6 years ago

    You are ignoring test failures in your plugin config in this line:

     

    <testFailIgnore>true</testFailIgnore>

     

    Actually, that's not quite correct, according to the documentation, this property determines if a test will continue running if it encounters an error:

     

    Continues the test run even if an assertion triggers.
Possible values: true, false.

     

    After your test runs, you can use the JUnit plugin to parse the results, and that will pass/fail the build.

     

     

  • JustinM89's avatar
    JustinM89
    Contributor
    6 years ago

    You are ignoring test failures in your plugin config in this line:

     

    <testFailIgnore>true</testFailIgnore>

     

    Actually, that's not quite correct, according to the documentation, this property determines if a test will continue running if it encounters an error:

     

    Continues the test run even if an assertion triggers.
Possible values: true, false.

     

    After your test runs, you can use the JUnit plugin to parse the results, and that will pass/fail the build.