Forum Discussion

sdeevers_starz's avatar
sdeevers_starz
Occasional Contributor
27 days ago

R-API 3.59.0 shows test case passed although there are failed test steps

During our evaluation of R-API Release 3.59.0, we noticed if a test case has failed steps, the test case itself is marked as passed.  This could lead to people thinking their tests are passing at a quick glimpse when in actuality, the test has failed.

Below is an example of a test with an SLA of 1000 milliseconds, which shows those failures in the "Transaction Log", yet the test case shows the tests passed.

The associated "Test Case Options" have the "Fail Test Case on Error", "Fail Test Case if it has failed Test Steps" selected, so something seems to not be working correctly. 

We are upgrading from R-API Release 3.53.0 to R-API Release 3.59.0 and we did not see this issue with R-API Release 3.53.0.

Has anyone else noticed this behavior with R-API Release 3.59.0?

5 Replies

Sort By
  • mnwill117's avatar
    mnwill117
    Occasional Contributor
    5 days ago

    We found that increasing the Max Results setting to be greater than or equal to the number of test steps in the test case accounted for all tests steps, and failed the test case if any failed. 

  • sdeevers_starz's avatar
    sdeevers_starz
    Occasional Contributor
    14 days ago

    Just checked the latest R-API Release, 3.60.0, and the issue of a test step failing and the Test Case and the Test Suite showing pass is still present :-( 

    This is core functionality that needs to be addressed as testers need to know when a test case or test suite has failed in the UI without having to drill into the "Transaction Log" after each run to see if something truly did fail.

  • sdeevers_starz's avatar
    sdeevers_starz
    Occasional Contributor
    21 days ago

    Just a follow-up on the above noted issue with R-API 3.59.0.

    In the current documentation for the "Default Test Case Options" the default value for  "Max Results" is 5.

    Making the change manually for each test case's "Test Case Options", "Max Value" to have a value of zero (0), would be an arduous task.

  • phuong_pham's avatar
    phuong_pham
    Occasional Visitor
    26 days ago

    Yes, I've also encountered the same issue when running my tests using ReadyAPI version 3.59.0. Previously, I was using version 3.54, and this issue did not occur.

    I found that setting Max Results to 0 in the Test Case Options seems to resolve the issue.
    Hope SmartBear can look into this and provide a fix soon.