Ask a Question

API Testing Mistake #4: Separating Load Testing from Functional Testing - Robert Schneider

Do you still run your functional and load tests separately? In the current episode of the 7 Common API Testing Mistakes series with Robert Schneider, we will learn that we don't need to isolate tests of different types from each other. Watch the video and share your use cases with us.




Robert Schneider

Robert D. Schneider is a Silicon Valley–based technology consultant and author, and managing partner at WiseClouds – a provider of training and professional services for NoSQL technologies and high performance APIs. He has supplied distributed computing design/testing, database optimization, and other technical expertise to Global 500 corporations and government agencies around the world. Clients have included, JP Morgan Chase & Co, VISA, Pearson Education, S.W.I.F.T., and the governments of the United States, Brazil, Malaysia, Mexico, Australia, and the United Kingdom.


Robert has written eight books and numerous articles on database platforms and other complex topics such as cloud computing, graph databases, business intelligence, security, and microservices. He is also a frequent organizer and presenter at major international industry events. Robert blogs at



Watch the previous videos in the 7 Common API Testing Mistakes series:

  1. Focus on the Most Typical Messages
  2. Using Hard-Coded Test Data
  3. Shielding QA Team From End Users




Today, we are going to talk about different types of testing and how to do that together with API.



Today's talk is going to be about a really common situation that we see when working with customers of all shapes and sizes, and that is keeping your functional tests separate from your security and load tests. Most organizations (with the exception of the very few that have leadership that really thinks ahead) will develop tests, a lot of times they're just unit test, but even the relatively few of the organizations that go beyond that and fully exercise their API testing tools to do functional tests as in a business process from start to finish as a functional test with maybe five to seven API calls with moving some data between the API's. That's a functional test. And a relative minority of organizations do that. But, to make matters worse kind of piled on top of that - even smaller percentage will take those functional tests which represent a business process of some sort, and then subject it to load and/or security testing.

In most cases, what we see is - there are separate groups for functional testing, separate groups, often outsourced, for load testing and for security testing. So what happens is they will bring in outside experts for load testing and for security testing. These people very often, not always, but very often, don't know the applications that the API's are designed to try to solve. So, what they will do is, in the case of security, they will come in and do the standard attacks on your endpoints, but they're doing those in isolation. In other words, one API call has nothing to do with a second API call, or a third API call. That's what you see on security.


On load, you'll see them come in, and they will do something similar. They will subject your various API endpoints to heavy volumes of load. And, they'll try to see whether this API or that API is breaking, or whether they are seeing excessive latency or things like that.


That's a problem in both cases. Because in the case of security, they are testing your API’s in isolation, they don't really know how your business process will behave when the process itself is placed under attack. Now, I'm not arguing for abolishing the security tests as they currently stand. And, the same thing with load testing - I'm not arguing for abolishing putting pressure on your endpoints to see if they will collapse under the load. But, when you take the hard work you've done in your functional tests and you extend that in a tool like ReadyAPI or SoapUI where you can with a click of a mouse say, ‘Turn this functional test into a security scan!’ or ‘Turn this functional test into a performance test!’ you're preserving all of the hard work you did in your functional test design. And, you were then subjecting it to a specific set of attacks and other weaknesses that you wouldn't have seen by going after the individual endpoints from security perspective.


It'll also give you a good indication of how the business process will stand up when you're subjecting it to load rather than just the individual end points. So, why not reuse the work that you've done with your functional tests in a security and load contexts? That's what we think is a much better practice, and it will find you problems, will discover problems earlier in the lifecycle, than waiting till you get to production. And, finding out that although, for example, the individual end points are perfectly fine with being subjected to load when you pull them together, and then a commonly used business process, guess what, it doesn't scale, it has problems. You wouldn't have found that if you didn't reuse the functional test.



This sounds reasonable. I suppose this is somehow related with the topic that we have discussed - using user scenarios in our API testing. It is an important topic. We just add some more tests, some secure tests to these user scenarios.



That's right. A lot of the things that we're doing in this series are really kind of tie in together.

The reason why people are not reusing their functional tests very often is a structural one. In the organizations in question a lot of times what they've done is they have a functional testing team and they have outsourced the security and the performance testing to specialists. And, we do not say that's a bad thing - that's a good thing. Because the outside experts live and breathe security or performance testing, but they don't live and breathe your functional user environment that you as a tester working inside the organization or as a partner with the organization know better than anybody else. So, we're saying these are additive types of things. Do not replace them, but add them to your mix, and you're going to have a much better chance of finding and fixing problems before they get to production and are harder to find at that point.



This makes things much easier. In this case, you have already set up this API testing, and you just add one more kind of an entity.

That was a very interesting mistake that we will need to pay attention to and try to avoid in our workflow. Great, thanks a lot!



Thanks everybody. See you next time, bye!


Next time, we'll review the next mistake. If you have any questions or suggestions about the mistakes, please leave your comments under this video, and we will be happy to continue our conversation online.


SmartBear Alumni (Retired)