Ask a Question

API Testing Mistake #1: Focus on the Most Typical Messages - Robert Schneider

All of us want to create reliable and effective API tests. We are here to help you with this! We are starting a series of interviews to analyze the most common mistakes that occur when testing APIs. We will tell you how to avoid such mistakes and change your testing methodology. Today, we will discuss why focusing only on the most common messages when preparing tests is bad.



Robert Schneider

Robert D. Schneider is a Silicon Valley–based technology consultant and author, and managing partner at WiseClouds – a provider of training and professional services for NoSQL technologies and high performance APIs. He has supplied distributed computing design/testing, database optimization, and other technical expertise to Global 500 corporations and government agencies around the world. Clients have included, JP Morgan Chase & Co, VISA, Pearson Education, S.W.I.F.T., and the governments of the United States, Brazil, Malaysia, Mexico, Australia, and the United Kingdom.


Robert has written eight books and numerous articles on database platforms and other complex topics such as cloud computing, graph databases, business intelligence, security, and microservices. He is also a frequent organizer and presenter at major international industry events. Robert blogs at


Next time, we will analyze another common API mistake and share the solution with you. Stay tuned 😉 Subscribe to this blog to always be in touch with the latest Software Quality Assurance news.




Let's start with a short introduction to the SmartBear Community.  



Sure, I've been working in the Middleware Infrastructure software testing API space for a long time. And, we've been a partner of SmartBear going all the way back to the days when SoapUI first came about. In the interim, the last 12 years, we've been teaching courses on the optimal usage of SmartBear’s excellent testing technologies like ReadyAPI and SoapUI. And, SmartBear continues to sell our products, and we’ve delivered these classes over the web and on site on every continent except Antarctica.



Nowadays, most of us want to create effective and reliable API tests. And, to do this, we will need to avoid making mistakes. Which mistakes are we going to talk about today?



Well, today we're going to be talking about the danger of testing only the most common messages and elements within a message, but this is again, as Tanya said, this is going to be part of a larger series of talks. In the 12 years we've been doing this, we've seen pretty much every API implementation that's out there, and they all seem to have very similar mistakes that they make and it's not because they're bad people lazy or anything like that it's because they're overworked. The APIs are becoming more and more important, they have less and less time to test them. Maybe, they're in agile environments. All of these things kind of contribute to this scenario, these scenarios that we see where people are putting APIs into production that are not thoroughly tested. And, the end result is, eventually, the user will find those mistakes usually at the most inopportune, bad time to do that. So, my hope is, in this series of talks, that we are very happy to be part of, we'll share with you some of the most common mistakes all of which by the way can be addressed by properly employing the SmartBear technologies.

Today, we're going to talk about not covering messages and elements within those messages, but we'll be talking about data-driven testing and integration with automation, all those good topics in the upcoming sessions.

So, not covering and not thoroughly testing your API only focusing on the most common messages that are out there and most common interactions with your API, I think, it's probably one of the most important things you can be doing. We see this all the time where if you think about the number of permutations on an API, let's imagine your API has ten elements that you transmit to it, and you've got to think about all the combinations of those elements, and what the data might be for each one of them. If you go look at a factorial calculator online it's kind of eye-opening to see just how many combinations of information you can transmit whenever you've got an API that needs that kind of data. And, a lot of times, what people will do is they'll use hard-coded data and normally test the elements that they think are most important. If you do that, there's a really good chance you're not going to be truly exercising the application logic on the back end.



And, will it bring some more load to the testing team? Will we need to create more tasks?



The good news about using technology such as ReadyAPI and SoapUI within ReadyAPI is that this may sound really intimidating like I've got hundreds of elements that I've got to transmit to my API. How am I gonna sit there and type this all in? The good news is you don't have to do it that way. There’s automation in the product that you can use to at least get you most to the way there.

Maybe of the 10 elements only six of them are relevant to your business. Other ones may be like email addresses and phone numbers and other types of data. You just need to put some information in those fields, and a lot of times people don't. They just transmit the fields that are relevant to them, and they leave the other ones blank which is really bad. Good news is you can use SoapUI and ReadyAPI to generate information for you that you'll transmit to the API. And, at least, get some coverage on that.

We're not asking you to come up with every possible city, or state, or country that you're transmitting in an API, but at least fill that information in, and you can very often send a packet of data some of which is information that's relevant to your business, other information is at least generated for you. The whole packet is now sent in. That's on the inbound side. On the outbound side, you should be thinking about - we need to look at the information that comes back and think about all these different possible scenarios that we have. The good news again, as I was saying, is that you don't need to be writing lots of different test cases for this, you simply need to be creating data that drives your API tests for these scenarios.

So, when properly testing API’s, you move away from a test developer to more of a test architect, and then, when new scenarios arise, you come up with the data that should plug in to the templates that you've created as part of your overarching testing strategy.



So, it sounds like we will need to create a template once, and, after that, we can use it in all the tasks.



Ideally, that's right. And, the products make it really easy to do that. This is not very difficult work. Thinking about it requires some effort, but once you've got your template, your structure in place, it's pretty much a matter of adding new scenarios as data rather than creating the next test case and the next test case. In fact, when we go talk to a customer, and they say with pride, ‘We have 10,000 test cases.’ I think, ‘Oh boy. That's a lot of test cases. You could probably get away with 100 test cases with a thousand different scenarios per test case, and data lets you do that.’ And, if you do all these things, and you properly cover your messages on the inbound and your responses on the outbound, you can have a much better possibility of catching errors early on rather than waiting for them to get to the user level.

Especially in this world of agile, it's physically impossible to sit there on your keyboard and type in every possible entry that you're gonna transmit to your API and do it 30 times a day. We'll talk about this in an upcoming interview - we're gonna need to be thinking about automation. Automation and data-driven testing, they kind of go very well together. And, data-driven testing on top of a complex API is the only way to achieve that. So, all these things kind of filter into each other and influence each other. But, again, we wanted to start this series of talks today by simply pointing out the complexity of most APIs, the importance of most APIs, and the reliability of an API can only be guaranteed by thoroughly testing all of the elements and the messages that you transmit to it and asserting information that comes back from it.



Thanks a lot! So, next time, we'll discuss and we will analyze another mistake and find the solutions to it. So, stay tuned. Community, if you have any comments, recommendations, questions, feel free to post your questions under this video, and we will be happy to continue our conversations online.



Till next time! Thank you.



SmartBear Alumni (Retired)