Ask a Question

API Testing Mistake #3: Shielding QA Team From End Users - Robert Schneider

How often does your software QA team talk with your end users? In the modern world, this is one of the most efficient ways of learning the actual user scenarios! Watch the new episode of the 7 Common API Testing Mistakes series with Robert Schneider where he explains this mistake in detail.




Robert Schneider

Robert D. Schneider is a Silicon Valley–based technology consultant and author, and managing partner at WiseClouds – a provider of training and professional services for NoSQL technologies and high performance APIs. He has supplied distributed computing design/testing, database optimization, and other technical expertise to Global 500 corporations and government agencies around the world. Clients have included, JP Morgan Chase & Co, VISA, Pearson Education, S.W.I.F.T., and the governments of the United States, Brazil, Malaysia, Mexico, Australia, and the United Kingdom.


Robert has written eight books and numerous articles on database platforms and other complex topics such as cloud computing, graph databases, business intelligence, security, and microservices. He is also a frequent organizer and presenter at major international industry events. Robert blogs at



Watch the previous videos in the 7 Common API Testing Mistakes series:

  1. Focus on the Most Typical Messages
  2. Using Hard-Coded Test Data




A lot of times, and it's not their fault, the testers are kept far away from the business users, the ones who are actually commissioning the API, and the ones who are going to validate ultimately that the API is working. A lot of times, we'll go to a customer site for a training class or a consulting engagement, and we'll talk to the testers, and the developers sometimes, too, by the way, and we'll say, ‘So, how much interaction do you have with the users?’ They say ‘none’ or ‘very little’, and then I say, ‘Well, okay, what about a specification? Do you have a spec on what the API is supposed to do?’ ‘No.’ Sometimes, they obviously have. If it's a web service they have the WSDL, or if it's a REST API they might have a Swagger or an open API document. But, a lot of times, the users are shielded from the testers and vice versa. And, that's a real problem for a lot of different reasons.


Probably, the biggest one is that the testers are essentially performing Quality Assurance on something they know very little about. They will look at the API, and they'll try to think about what the API might want to do, but they don't really know. And, this can be solved quite easily.


Many times, the end users and testers are in very different locations. A lot of times, the testers in fact are in an outsourced organization, so they don't really work in the same place as the end users, but, truthfully, a lot of those things can be solved by just doing things like what we're doing now. Internet communication, set up a Skype call, or a Zoom meeting, or whatever and talk about what it is that they're trying to achieve.



Okay, so, yeah, I know that now many companies do some kind of testing reviews where they're talking with testers in order to understand what they're doing in order to get the information that can be used so I think that it can be useful in this case, as well.



Absolutely. And, a lot of times, testers can be shy about asking the user specifically what they want. And, sometimes, the users don't know. But, that's another story for another time, which is kind of a sad thing, but, hopefully, the users should have some idea about what it is they're trying to achieve with this API the people are going to be testing. But, obviously, there are some serious ramifications about doing this.


What we see happen when people are separated from their end users is they will attempt to guess at what the API is supposed to do. But the way that this manifests is they will very often instead of writing comprehensive functional tests, they will fall back to the most narrow unit test that they can do. So, they'll say, ‘All right, I don't know what this API does, but I do see that there's a call to look up some order.’ Let's say, it's an e-commerce application. I'm gonna look up an order using an order ID, and I'll see what it brings back. Great. But what you don't really know because you're not talking to the users is that this is part of a much larger process of the preparing the order for shipment, for example, business process. So, not only do we have to look up the order, but we have to do some validation of the address, and validation of the payment method, and validation of the shipper as part of a larger functional test. And, if the tester or the developer doesn't even really know that they're gonna focus on each of those steps by themselves with no continuity across the process. That's not optimal because you don't really know if things are going to flow very smoothly if there's going to be connection issues between pulling the data back from one API call and feeding it to another. So, really, you want to be understanding the entire process as best you can as early as you can in the design phase.



Yeah, so, it sounds like in order to create test scenarios, you will need to talk with the users. Once you create these test scenarios, you can start creating tests because you have all this kind of prepared stuff, so you will only need to create tests to cover all these scenarios.



That's right. I mean part of this also a lot of times since you don't speak to the users not only do you not know the application or the scenarios that you're supposed to test, you don't know the errors, how errors should be handled. Continuing the example of the shipments - what if there's a bad address, what do we do then? If you're only doing unit testing because you don't know what the API is supposed to do, you're going to miss a lot of these nuances, and, of course, what's going to happen is when you go to move this into production because you said it works, and the first time it hits a scenario maybe it's one in a hundred, but if it's a large application, you'll hit that scenario in the first hour of production. You're gonna have to go back and reverse engineer what the problem was. It's going to make you look bad, it's going to make the users unhappy, and it could all be prevented by a brief conversation or periodic brief conversations to truly understand what it is that they want.



Okay, great. Thanks a lot, great suggestions. So, community members, do not be afraid of talking with your end users because you will get a lot of information from them that will help you create wonderful reliable and successful tests.


Next time, we'll review the next mistake. If you have any questions or suggestions about the mistakes, please leave your comments under this video, and we will be happy to continue our conversations online.




SmartBear Alumni (Retired)