JUnit Reporting of individual runs in a data driven test

Hi Mahesh,

you should get an aggregated report on all failed TestSteps within the loop of your TestCase (although it may be hard to read).. Are you using 2.5 beta or 2.0.X release?

regards!

/Ole
eviware.com

Hi Ole,

I am using the Soap UI Pro 2.0.2

I do see the aggregated report of all the failures in the JUnit result. As you pointed out, it is hard to read if there are many failures. 

For better readability (especially for the business analysts who need to investigate the failures), I have now started dumping some of the key data elements from the request as well as the actual vs expected values from each assertion for the failed iterations into a spreadsheet. 

If there's a way to report individual failure details of each failed iteration of the loop inside the test case, that would be perfect. It would also provide for better test reporting, I think ... instead of reporting that 1 Test Case was executed and whether it was successful or not, it would provide visibility to each iteration of the test case, and also provide better metrics like the number of failures, time for each iteration, etc. 

Thanks
Mahesh

Hi Ole

I worked around this reporting problem by setting up a custom listener (cloned from JUnitReportCollector), and it solves my problem. In my custom JUnitReportCollector, I am adding to the reports map in the afterStep method, and I do this only for the 'Request' test step(s). In the afterRun method, I save the report files (Junit report xmls). Since my SoapUI is running under Hudson, I let Hudson consolidate and do the JUnit reporting & trending.

One other advantage of the custom report is that now I have the ability to print out more information in the 'Sysout' section of the JUnit report (e.g., properties). Regarding properties, is there an API to retrieve all properties used in the test case / test steps ... I would like to write a generic 'print properties' so that my custom reporting is not tied to this particular test case alone.

I did run into another problem. I am now seeing some out of memory errors. My request / response XMLs are fairly large. I find that these OOM problem starts happening only when the number of  failures increases. I have bumped up the Heap to 1024M, but that just delays the OOM error. I saw your blog on the Max Results option in the Test CAse options. Is that option available in 2.0.2? Is there any other way in 2.0.2 to release all test results for prior iterations of the data source loop?

Thanks
Mahesh

Hi Mahesh,

You can manually call discard() on TestStepResult objects that you no-longer need, soapUI 2.0.2 will automatically keep all errors so this might be an option (discard after you have reported them..)

sounds reasonable?

regards,

/Ole
eviware.com

Hi Ole

The discard() call works. Thanks for the suggestion.

Thanks
Mahesh