Performance testing simultaneous requests

[nastester]

This is probably a dumb question but how can I go about performance testing thousands of simultaneous calls at once to observe how the system handles that?

The performance/load/stress tests I've done with ReadyAPI in the past have been about number of requests over time with VUs. 

In this instance, I don't want say an hour long test run with the same request over and over, I want do test simultaneous calls.  How can I go about this? Couldn't really find anything in the docs unless I missed it.

[KarelHusa]

@nastester,

the VUs are parallel clients of your API. You can set the VUs and start the performance test. 

With ReadyAPI Performance license, you can set the VUs number higher.

What are you missing? Do you want to make different request data for different VUs?

[nastester]

I guess what I want is to process a test that executes X amount of requests and then stops. 

So would I just set X amount of VUs and set max targets to 1 or something? How can I ensure that all those VUs are actually running the request? Or is there a better way to accomplish this.

For the scenario I am trying to replicate, it is not realistic to process thousands of requests over and over for an hour test run for instance.

[KarelHusa]

@nastester,

I recommend first defining a scenario, what you need to test.

Do you only need to send N requests and see, how the app handled them? Then number of VUs does not matter; you only need to specify the total number of targets.

Would you like to simulate real load on your API? Then VUs matter as they simulate real users consuming API in parallel.

ReadyAPI ensures different VUs handle the requests. You just need to set up the number of VUs.

[nastester]

@KarelHusa 

The scenario I am trying to create a test for is this:  An external application calls our API with 3000 concurrent requests. 

I am trying to replicate this scenario and I think I am missing something fundamental. These tests are seemingly set up as time based or max target based. 

I just want to simulate 3000 requests at once. 

[KarelHusa]

@nastester,

sending 3000 concurrent requests means creating 3000 VUs, each sending a single request.

I am not aware of the background and purpose of this test, but starting up 3000 clients (VUs) from zero at the same moment is rather unusual. Most of the systems would be surprised by such a peak unless designed for it. Maybe the load strategies description could help you.

[TNeuschwanger]

Hello @nastester 

Please be aware that the maximum number of concurrent virtual users your test can simulate is limited by your ReadyAPI Performance license.  Do you have licenses for the 3000 you desire?  You can make the number of virtual users whatever you want if you can pay for them.  🙂 

The company I work for is looking into purchasing a few thousand virtual users, but the license for them gets expensive fast and we come to a point where we have to decide if it is worth it or not.  How often do you need to performance test?  If if was some sort of regression and you did it all the time, then the money would seem to be worth it.  If you are just curious to see what the limit is on your API, and only tested one time with big virtual user numbers that seems like poor stewardship of your resources.

Do you have available agent computers to distribute the virtual users to?  The documentation I have found says the computer that ReadyAPI is running the performance test from can support up to 500 virtual users.  For every 1000 beyond that, you need to distribute to additional computers with a performance agent running on them.

Regards,

Todd

[KarelHusa]

@TNeuschwanger,

I have not found the 500 VU limits in the documentation, only very vague wording "This depends on the license". Multiplying every 500 VUs by 6500 Eur seems unbelieavable. I am long-term fan of ReadyAPI and SoapUI, but such prices are not aligned with the benefits. Even 6500 Eur for unlimited VUs would be way too much.

JMeter is free, and offers unlimited number of VUs with the option to run tests on multiple machines. JMeter is much stronger in performance testing in terms of functions and simulating real load. JMeter also supports Groovy scripting.

The strength of ReadyAPI load testing is in re-using the functional tests. And I appreciate launching a load test with two clicks. But, it works for very basic tests only.

When I design a real performance test, i.e. simulating real load I need to create a dedicated test anyway. In summary, JMeter is far cheaper and more efficient for such purpose and you don't reach almost any limits in your scenarios.