Ask a Question

API Load Testing Mistake #1 to Avoid: Separating Load Tests from Business Process - Robert Schneider

SmartBear Talks are back! And, we continue talking about API testing. You might remember that we already learned about 7 Common API Testing Mistakes we need to avoid. In this series, we will focus more on API load testing!


API performance testing becomes highly-demanded in all companies. It's very important to make sure how each single API works, as well as how they work under a different load. Many companies are trying to do API load testing by using the same approaches they use to test single APIs. In this video, we will talk with a world-known API expert, Robert Schneider, about the importance of including API load testing scenarios into a business process.

Watch the video or read transcript below:


Robert Schneider

Robert D. Schneider is a Silicon Valley–based technology consultant and author, and managing partner at WiseClouds – a provider of training and professional services for NoSQL technologies and high performance APIs. He has supplied distributed computing design/testing, database optimization, and other technical expertise to Global 500 corporations and government agencies around the world. Clients have included, JP Morgan Chase & Co, VISA, Pearson Education, S.W.I.F.T., and the governments of the United States, Brazil, Malaysia, Mexico, Australia, and the United Kingdom.


Robert has written eight books and numerous articles on database platforms and other complex topics such as cloud computing, graph databases, business intelligence, security, and microservices. He is also a frequent organizer and presenter at major international industry events. Robert blogs at


Also, Robert is an author of 7 Common API Testing Mistakes video series.


I hope you like the video! If you have any questions, please post them here. The series contains 5 videos that will be posted each Monday of August 2020. Subscribe to the SmartBear Community Matters Blog to get the news from us first!





Hey! I'm welcoming you in SmartBear Talks where we are discussing exciting and trending topics with software testing experts. We are returning to you after a short break with some great news!


We are so excited to announce a new series of interviews. You might remember the ‘7 Common API Testing Mistakes’ series that we did last year with Robert Schneider, a software testing consultant from WiseClouds, a strategic SmartBear partner. And, this time, we will continue talking about API testing with Robert, but we will mostly concentrate on API load tests.


Hi Robert! How are you today?




Hi! Hello everybody! It's great to be back especially during these crazy times that we're all living in. I hope everybody out there is well and staying safe, and I am really happy to be back to talk about the next series of topics that we will be covering in these interviews. Hopefully you're finding them interesting.


We're going to be talking about some of the most common API performance testing problems that we're seeing. We talked last time about some of the most challenging API testing problems that we've seen, now, we've taken it a different level, different place, and we're talking now about performance testing.


Over the course of the next five interviews which will be done by myself, as well as my colleague, Damion White from WiseClouds, who's a performance architect, we're going to be talking about things that we've seen in working with customers.


To briefly summarize what we will be talking about: today, we're going to talk about the dangers of not fully testing a business process under load. The next interview, which Damion will be doing, will be about not calibrating your load test upfront to factor in the number of business users versus virtual users. That's a common problem we see. I'll be describing later on in another interview why a lot of people don't go through the trouble of testing multiple load generation scenarios strategies for things like that. Damion will follow up by talking about some of the most common security challenges that we see when testing especially GUI-based APIs, or APIs that have a GUI front end, a very common kind of scenario. And, I'll close up this series by talking about what happens when we don't factor in external API calls into our test plans, and that's a very common problem from a performance perspective especially, which is what this series is about. So, I’m quite excited to talk to you guys about these things.


So, shall we get started talking about the dangers of not performance testing a full business process, what do you say?




Yeah, I look forward to listening to your opinion! Please share your expertise with us - what mistakes we can make during API load testing and what we need to pay attention to in order to avoid them!




All right. So, I'll begin. From my perspective, an API equates to a micro service. I know you can get huge battles between architects and developers saying: ‘This is an API!’ ‘No, it's micro services!’ From our perspective it's the same thing. It's distributed software components that work together to achieve some kind of common goal. With that said, let's start by talking about the dangers or the common problem we see of not fully performance testing a business process.


If you remember from the last interviews that I did with Tatyana, which actually was a year ago believe it or not, so in that time, I would like to say things have improved in the industry. Unfortunately, we still see a lot of the same problems. One of them is you have these business processes, and it's hard enough to simply do functional tests from the API perspective on a business process. Most people, when they're testing their APIs, they are making single endpoint calls, one after the other, almost like a unit test on the API. They select few, take those API calls, and they batch them together into a functional test and execute the functional test as a cohesive unit. And, even a smaller number of those people go through the extra trouble of taking that functional test and subjecting it to performance or load testing types of scenarios. And, the problem behind that is a lot of people are taking their individual API endpoints and subjecting them to load, as well. So, they're not testing the business process, but are instead simply hammering on each individual endpoint with large volumes of data to see where it will fail. And, that's fine. We don't say: ‘Don't do that.’ Go ahead and do that, but what we think is you also want to be testing your full API process. And, I'll give you an example - a full business process on the APIs.


So, imagine you have an e-commerce site, and on your e-commerce site, let's say, the average user transaction you've got, maybe, 10 searches of the e-commerce site. Maybe, three items added to the cart. Maybe, one click of the ‘Checkout now’ button to take payment and such. So, it's a 10 to 3 to 1 ratio. If most people are not testing the API with 10 times the load on the lookup API calls and three calls, let's say, for putting things into the cart, and one call for testing the output, so if you're not really testing the full business process, you miss all of those ratios. Now, you can try to reverse-engineer what those ratios might look like and fill it in in some way, but why not just use the appropriate full business process and place that under load? Tools such as ReadyAPI and LoadUI let you do that very easily. The same functional test you design in SoapUI, you bring across as a load test, and you can then spin up as many instances of that load test and truly test the business process.  




I suppose this is somehow related to the load testing scenarios that testers should pay attention to.




We're going to talk about that in one of my next sessions with you guys. It all links together. So, everything that we describe in this series, think of it as the building blocks that all kind of work together to come up with a cohesive result.


So, business processes are such an important thing, and, by the way, I won't underplay the challenges here. A lot of times the business users can't even tell you what the actual use scenario is going to be like. So, as testers, or developers, or architects, whatever, it's kind of hard for you to figure out, but there are ways to do it, at least, make a guess at it. And, again, to be very clear, we're not saying do not test the individual endpoints under load. For sure, put them under stress, see where the smoke starts coming out of the machines, but try to take these things and put them together into the functional test. And, by the way, if you're doing that anyway, it's just really simple to turn that functional test into a load test. It's the same process except, now, it's being run by hundreds or thousands or millions of virtual users or virtual arriving processes to see how the actual business process will perform under load.




I cannot say that you can isolate APIs and test them as they are, but if you are testing them at the moment it's okay. You can continue testing, but it is very important to understand the order. For example, if your test is calling one API, and after that, it is calling another API, and, again, another API - you need to test the full path of these APIs.




That’s right. Because they don't operate in isolation, they operate as part of something larger and, that's why, it's so important to do these kinds of functional test - number one. And, number two - to do the functional test under load. You'll get such a better picture of what kind of performance and responsiveness you can get in run time. Again, don't throw anything out. This is an additive type of thing. So, for sure, keep testing your endpoints and see where they fall over, but try to gather these things together into an organized logical workflow because you need that anyway. And then, place that whole thing under loading. ReadyAPI is superb at doing that. It's very straightforward, and it really can help you uncover some hot spots.




Okay, great. Thanks! That's great advice. Let's move forward to another one and another mistake. We will talk about it very soon, so stay tuned.




All right! Everybody, stay tuned, we'll be back with more!

SmartBear Alumni (Retired)