Forum Discussion

JochemNS's avatar
JochemNS
New Contributor
3 years ago

Difference between TestCaseRunner Summary and Reporting

Hello,

 

When running tests via the testrunner we see differences between the TestCaseRunner Summary and the generated reporting.

 

In some cases the TestCaseRunner Summary fails the run, but the generated reporting does not give any errors (which is expected).

 

Is there something that can explain this behavior?

 

The project that created these results is very heavy on scripting and creating testcases on runtime based on input from an excel file. Its also very modular, so some testSuites are disabled, but testcases within those disabled testsuites are being called by other testcases. 

 

An example:

 

 

 

Assertions
Function Tests
Reporting
Test Results
  • nmrao's avatar
    nmrao
    Champion Level 3
    3 years ago
    That's strange. Please have look in the logs for more information why it is showing like that.
  • JochemNS's avatar
    JochemNS
    New Contributor
    3 years ago

    Thanks for the reply.

     

    What should I be looking at in the log? I know that there is an error being thrown in one of the groovy scripts. But if that triggers a failed Testcase in the testrunner summary, I would expect it would also trigger a failed testcase in de report. Even if we fix that error and we have a "Green" run, all counts (of testSuites, testcases, teststeps and assertions) are still way different in both reports. It seems like that the testrunner uses a different "source" than the report for the summary... 

  • nmrao's avatar
    nmrao
    Champion Level 3
    3 years ago

    I said in the context why two different results; I mean basically both should match.