This is probably a dumb question but how can I go about performance testing thousands of simultaneous calls at once to observe how the system handles that? The performance/load/stress tests I've don...
Please be aware that the maximum number of concurrent virtual users your test can simulate is limited by your ReadyAPI Performance license. Do you have licenses for the 3000 you desire? You can make the number of virtual users whatever you want if you can pay for them. 🙂
The company I work for is looking into purchasing a few thousand virtual users, but the license for them gets expensive fast and we come to a point where we have to decide if it is worth it or not. How often do you need to performance test? If if was some sort of regression and you did it all the time, then the money would seem to be worth it. If you are just curious to see what the limit is on your API, and only tested one time with big virtual user numbers that seems like poor stewardship of your resources.
Do you have available agent computers to distribute the virtual users to? The documentation I have found says the computer that ReadyAPI is running the performance test from can support up to 500 virtual users. For every 1000 beyond that, you need to distribute to additional computers with a performance agent running on them.
I have not found the 500 VU limits in the documentation, only very vague wording "This depends on the license". Multiplying every 500 VUs by 6500 Eur seems unbelieavable. I am long-term fan of ReadyAPI and SoapUI, but such prices are not aligned with the benefits. Even 6500 Eur for unlimited VUs would be way too much.
JMeter is free, and offers unlimited number of VUs with the option to run tests on multiple machines. JMeter is much stronger in performance testing in terms of functions and simulating real load. JMeter also supports Groovy scripting.
The strength of ReadyAPI load testing is in re-using the functional tests. And I appreciate launching a load test with two clicks. But, it works for very basic tests only.
When I design a real performance test, i.e. simulating real load I need to create a dedicated test anyway. In summary, JMeter is far cheaper and more efficient for such purpose and you don't reach almost any limits in your scenarios.