Ask a Question

API Load Testing Mistake #2 to Avoid: Not Calibrating Virtual Users to Real Users - Damion White

A huge part of your strategic plan for load tests built into your app development business processes is understanding and ensuring that the user profiles or the actions that your simulated users take in any testing environment accurately depict what happens in a real-world use case. That's why you need to be very careful with setting up virtual users in your load tests. We will investigate possible issues with Damion White, software testing consultant from WiseClouds, in this video (find the transcript below):


Damion White

Damion White is a seasoned technology professional specializing in developing, testing, integrating and implementing strategic, web and software-based solutions for customers across a wide range of industries and business applications. Most recently, Damion joined the team at WiseClouds – a provider of training and professional services for NoSQL technologies, high performance APIs and GUI architecture – as a lead application performance architect, specializing in GUI performance. He has worked extensively across SMBs, mid-market and enterprise businesses and has been tapped to engage in highly regulated industries for clients such as Pfizer, HP, Intel, Sanofi-Aventis and Lenovo. This has given Damion unique perspective and insight valuable to clients whose businesses emphasize data privacy and governing protocols like HIPAA, PCI, GDPR & FINRA compliance.


Presently, he resides in Austin, Texas with his wife and two daughters. As a former two-sport NCAA Division I athlete, when he’s not producing and recording music, Damion enjoys spending a portion of his spare time coaching high school football and basketball.


Connect with Damion via Twitter or on LinkedIn.  


I hope you like the video! If you have any questions, please post them here. The series contains 5 videos that will be posted each Monday of August 2020. Subscribe to the SmartBear Community Matters Blog to get the news from us first!

Watch the previous video of the series:

  1. API Load Testing Mistake #1 to Avoid: Separating Load Tests from Business Process




Hello! I'm welcoming you in SmartBear Talks, and, today, we will continue talking about the major mistakes you need to avoid when you're creating API load tests. During the previous video, we discussed the necessity of including load testing scenarios into a business process, and this is the second video of this series.

Today, we're going to talk about the virtual users, and why virtual users are not real users. Damion White from WiseClouds, a colleague of Robert Schneider, will help us understand this topic better today.

Hi Damion, how are you today?



Hi Tatyana, I'm doing very well. I'm very excited to be here. How are you?



I'm good, thanks. So, please explain to us what we need to pay attention to when we're talking about virtual users.



Absolutely. So, as you led into this you hit the nail right on the head mentioning that you need to have a strategic plan for load and UI API load tests built into your app development business processes. A big part of that is understanding and ensuring that the user profiles or the actions that your simulated users take in any testing environment accurately depict what happens in a real world use case.

So, for example, many times, we don't have good connection or our business units, lines of business exist in silos, and it becomes difficult for development teams to get access to accurate user data that will instruct how we architect performance and load testing. However, it's very-very important that we understand anticipated user metrics, and how users behave in an application. Or, else, we run the risk of overshooting or underestimating what the actual load or stress that will be put upon an application might amount to in the real world.



Do you mean that we need to have some predefined delays because when we setup virtual users, they can just access the API during a tiny amount of time in milliseconds or whatever, but it's practically impossible that in a real situation real users will get access to your API so fast? And, that's why, I suppose, there should be some delays in order to simulate that.



You couldn't be more correct. And, the way we in the load testing world refer to those delays that you mentioned is we call those ‘think times’. So, imagine anytime that you access a web or desktop application that involves completing some activity or various numbers of activities, you will often stop intermittently. You may get up and go grab a donut or a cup of coffee, the phone may ring, whatever the case may be, real users in the real world don't just simply bang on the APIs or breeze through the user interface elements. So, it becomes the business of our data analysts, our telemetry folks, and our app insights people to deliver what those think times ultimately amount to. And, you know, it is oftentimes something that is not available.

Think about an e-commerce company or a retail business that wants to get into e-commerce. If you don't have any means with which to anticipate user traffic before embarking upon an app development process that involves that, it becomes very hard to calibrate those virtual users. And, in many cases, we end up producing or architecting load tests that are greater in terms of the amount of traffic that we end up passing through in concurrent virtual users because everything is happening so fast and in a condensed manner that it doesn't actually reflect what a real user is going to do. And, that's problematic for a number of reasons.

Number one - it overstresses the application. It creates false negatives in terms of test failures, but conversely, it also runs the risk of not exposing those areas where there may be holes. If, for example, there's a bottleneck where users tend to take a larger amount of think time, and then return to IE, for example, checking out of a cart. If there was a number of elements that directed us to continue shopping in an e-commerce setting, but we did not understand how users typically behaved once prompted to take those steps in their journey, it becomes difficult to simulate that in a load test. And, those difficulties translate to apps that don't perform well in the real world upon release and/or extended development processes.



Thanks for this explanation! This was the second mistake you need to avoid when doing API load testing. Stay tuned, we will continue posting more mistakes you need to avoid.

Thank you Damion for this explanation!



This has absolutely been a real pleasure. Thank you for having me Tatyana. I look forward to our next session.

SmartBear Alumni (Retired)