[Day 3/12] How to Start With API Load Testing



In SmartBear's August 2017 customer survey, our customers told us that performance is a better measurement to determine API success than a usage metric, like the number of subscribers. Of course we need to provide our customers the right functionality but in the end, performance matters.


61% of our survey respondents claimed to be load testing their APIs. From countless conversations with your peers, I believe that number would be much higher with a few guiding principles.


A great place to start is building your API with a performance SLA in mind. This is equally important for internal or externally facing APIs, as well as services that applications sit on top of. What is an appropriate response time for customers of your APIs? Those APIs may be buried under layers of technology that also add precious milliseconds to the overall user experience. If my guitar tab application is making a call for the audio file location, calling for chord definitions, and at the same time calling for the song tab I should ensure I am testing performance of each of those components and understanding their overall impact on the application as well as the bottlenecks each service has.


We need to treat our APIs as first-class citizens and test our applications all the way from the front end user experience, through all of the services that are being called in our technology stack. Don't forget about the impact of those pesky 3rd party calls either. It's important to understand where the bottlenecks exist so that you can plan for peak traffic or normal traffic patterns. What if your chord definitions are spiking the CPU well above the CPU utilization for a tabs call. The chord definition API may need to have caching enabled, or memory optimizations applied.


Additionally, are you testing based on what your real end users are doing? For example, if your guitar tab API traffic is mostly users retrieving tabs to display, it wouldn't make sense to load test the add tab functionality at half of your load test. If your API is being leveraged a certain way, so should your API load test in order to closely mimic reality. 


For more tips and tricks:


It's your time to participate:

  1. Does your company have SLA's associated with services?
  2. Do you deliver APIs directly to your customers (internal or external)? What is your strategy to ensure they scale to meet your customers' needs without sacrificing performance?
  3. What issues have you faced in trying to start up performance load testing? How are you overcoming or how did you overcome those hurdles?


12 Days of Software Quality Rules:

Post your comments and give Kudos for comments you like. Tomorrow, we will announce a winner of Day 3 - a community member with the biggest number of Kudos for his/her reply. 

12 Days of Software Quality Schedule

Community Hero
Does your company have SLA's associated with services? We do, but they are not fully fleshed out or known yet. We recently acquired LoaUI as part of ReadyAPI and I am working on learning how to use that so as we get more and more SLA pieces defined I can make sure we adhere to them. Do you deliver APIs directly to your customers (internal or external)? We have an environment that is available external to us for the first line of customer testing. This usually takes places before a delivery, so we can get feed back before the lengthy delivery process to ensure our customers will be able to get the desired functionality out of our product. What issues have you faced in trying to start up performance load testing? How are you overcoming or how did you overcome those hurdles? Right now it's really learning the various ins and outs of LoadUI and how to work with a Write-based service and serving up data that meets the uniqueness rules without bogging down my computer. I've been able to successfully load test a Read service. For my load tests, I started using a pure groovy-script DataSource, since I care less about the data being written and care more about how much is being written and how quickly.
New Contributor

I am not Load testing my applications but we are going to in the near future. The above provided information can help us to do the pilot of Load Testing.

@msiadak - Thanks for sharing! Once I have an initial load test, I will be defining my SLA's and use those to articulate to upper management how we're doing with performance over time (% of SLA being met). Red/green seems much less complex than explaining complex performance scenarios.


Is the external environment you mention for the internal team to test on, or is it something your customers can interact with? I really like the idea of a customer sandbox. 


Good feedback on the groovy-script DB. That would be beyond my current abilitySmiley Happy

Community Hero

@JamesLunan: Glad I was able to share some useful information! Red/Green testing is definitely a lot less complex, and I am figuring out the transition to performance-based scenarios as well. I'm going to be coming up against that real quick here I think. As for the external environment, the internal team can (and does) test against it to verify installs/maintenance windows/etc, but we try to keep it a "clean" sandbox for customers to interact with. The internal team limits our "test" data being put out there and we try to purge what we do to keep the customer's data sterile and intact. 

Frequent Visitor

Does your company have SLA's associated with services?

-- We do have SLA's for overall response time that can be taken by bunch of services. In our system we have home page, we have various services

Which are invoked at the login time? Calling login service to player balance check, session creation and many other services. Once's player reach to final page where user have click on game to launch game and play a game (all done through service invocation).

We have created an end-to-end flow for such type of scenario's and generate load Test using Soap UI Pro.

We used Strategy like Simple and Fixed-Rate for our Performance Analysis. We do have groovy scripts which basically run the test cases conditionally for our performance, additionally monitoring these services on the server monitoring tool for the resources and max/avg time taken by single API.

Monitoring Server Help's us to say we are good.


Do you deliver APIs directly to your customers (internal or external)? What is your strategy to ensure they scale to meet your customers' needs without sacrificing performance?

Overall performance of system should not be impacted in one release that is our goal. if Some release is going on after 6 month then we do performance analysis like I mention in first step, if server monitoring say that it is because of modification in this particular service, we look for fixing that service by indexes if DB issues, even code fixes are done for such cases.


What issues have you faced in trying to start up performance load testing? How are you overcoming or how did you overcome those hurdles?

Setting up services flow , Groovy script code to do conditional calling , making platform similar to real environment - which service called most, should be called more time for performance analysis and technique variation(planning time takes more time then implementation).

Community Manager

Thanks everyone for a great conversation. Feel free to continue it in the future. However, rules are rules, and I have to announce a winner. Today, I want to congratulate @msiadak for earning the $25 gift card! Way to go, Mathew! We will publish one more article about the Load testing - check out our schedule.


Today, we've just posted a new article to talk with you about parallel testing. Join the conversation:


 @testhat thanks for sharing!



That's great that you've taken SLA's all the way through a transaction. I don't think our team is there yet but my goal is to take a business case and collect all the API calls involved and test those in order to meet an SLA. I like the fact that you've built 'no performance decrease' into the build process and release criteria. 


That's interesting that you have conditional load tests. What is the driver for that? Are you changing the load tests based on which parts of the code were updated?


Users online (91)
New Here?
Join us and watch the welcome video:
Join the September Hub-bub to show off, learn and win