LoadTest Statistics

Definitely..  already on our list..

thanks!

/Ole
eviware.com

Ole,
Would it be possible for you to tell me what kind of improvement are on the works in terms of load testing so that I can suggest more if some are not listed.

Thanks
A
li

Hi Ali,

of course! here comes a list:

1) Add an option to recalculate statistics with every teststep
-> currently statistics are recalculated first when a testcase finishes, which can be "annoying" with data-driven testcases which don't finish before all data has been processed.

2) Add a "Runs per Thread" limit option
-> The current "Total Runs" option is obviously confusing for many users initially, this option would be more in line with other tools

3) Change the gathering of statistics to be configurable as follows:
- It will be possible to add an arbitrary number of "collectors" to a load-test, for example a "Save-raw-data-to-db" collector, or collectors corresponding to the current diagrams and error log (which will be improved and refactored to be default collectors instead). This will remove the current "memory-consumption" of the loadtest-engine (or at least make it configurable) and also provide a foundation for custom collectors and printable report content (the same collectors will be used to create reports)
-> So for example your previous request to improve the LoadTest Assertion reports would fit within the "LoadTest Errors Collector" which would include more detailed information on which assertions that resulted in which error..

4) The possibility to save loadtest results (and testcase results) to monitor  and compare results/improvements over time

5) Exporting of statistics will be in conjunction with the collectors mentioned above

6) In the long run we would like to add more "enterprise-like" loadtesting functionality, like distributed load-testing and collection of system-statistics (memory consumption, etc..) during loadtests. Currently this can be partially achieved with the pushtotest testmaker framework which has integrated support for soapUI tests.. (check out http://www.pushtotest.com/)

Please don't hesitate to suggest more ideas and/or challenge the above!

kind regards,

/Ole
eviware.com

Hello Ole,
Regarding the writing of load test statistics to a database. What is your vision on accomplishing this. I mean will we have to host a database somewhere, who will define the database schema, what kind of information do you intend on pushing to the database.

What is the soapUI release that will likely have this much needed feature.

One of the ways that some of my other friends do at their company is that they have defined a database schema where information such as response times, errors, #users, as well as performance data such as CPU, IO, SQL counters etc is written.

They create automated test which fire off during continuous integration and generate reports automatically so that performance related problems are detected fairly early.

Having soapUI write loading testing data directly to the database would be extremely helpful.

Ali

Hi Ali,

thanks for your follow-up. soapUI 1.8 is likely to have improved reporting / exporting functionality for LoadTests.. but 1.7.6 will actually add a possibility to plug in custom listeners (written in java or groovy) into various events "inside" soapUI, including the execution of a LoadTest.. so if you need this "desperately" we will try to help you create your own plug-in for this until it is supported by the UI..

Regarding the database schema, when we integrate this you will likely need to follow some pre-defined schema, with the above mentioned possibility you will have full freedom to define your own schema/persistence mechanism.

stay tuned for more info on this together with the 1.7.6 release next week.

regards!

/Ole
eviware.com