[Day 2/12] Testing Frameworks - What I Should Know



Understanding of a testing framework starts from the understanding of the task that a QA person faces. After determining that an application needs to be tested, the high-level task for the tester can be described as "to test". However, the simple verb - "to test" - does not explain what this means and doesn’t cover the amount of work that stands behind. Every QA engineer that needs "to test" an application has to find their own solution to the task. The testing framework that is created during this process is the basis of this solution.


So, a testing framework is a system that helps accomplish the task of application testing. I think that the minimal set of elements the simplest framework contains is as follows:

  1. A test.
  2. An environment to execute the test.
  3. An analysis of the test execution results.


If we are talking about TestComplete, this is a TestComplete project with at least a single KDT or script test (#1). A tester runs the test on their PC (#2), and the log of the execution (#3) is saved to the project so the tester can analyze it. Of course, this simple scenario is not scalable enough and won’t be sufficient for many QA teams. So, let’s think about the way to improve every part of this simple framework.



Apparently, a single test is unable to confirm the quality of an application. Usually, there is a set of tests that a tester needs to manage. There are a lot of questions that could arise here:

  • How to organize tests? In TestComplete, you can use test items, multiple projects or projects suites.
  • How to work with objects within tests? For example, you can use the standard object tree provided by the testing tool or build a custom object repository that will access objects using scripts.
  • How to describe test steps? This can be a KDT visual test, a script, a custom Excel sheet, or one of the BDD frameworks.
  • Where to get data for tests? It is possible to use Excel sheets, database tables, randomly generated data, etc.
  • How to store tests? Tests can be stored in a shared location or a source code control system.



Usually, a QA team needs to check whether their software works without problems in all possible environments (e.g., operating systems of different bitness, web browsers, mobile platforms) having different security permissions, user roles, paintings on a mousepad etc. When searching for a way to handle this stuff, a tester can consider a variety of options:


Managing different test environments is related to another problem that also needs to be solved: how to run tests in these environments. Here, in addition to custom solutions, there is a bunch of ready-to-use options, as well:

  • Using the built-in features of the testing tool (e.g., Network Suite in TestComplete).
  • Utilizing CI systems used by development teams: TFS builds, Jenkins, Bamboo, etc.
  • Using test management systems like QAComplete, HP QC, etc.


Results and Analysis

After getting results of a single test execution, a tester must decide what to do with them, and there are a lot of funny things that the QA person can do:

  • Save the results somewhere locally.
  • Publish the results somewhere on the Internet or a local website for quick access.
  • The framework can send results via email or other channels to anyone who wants this information.
  • Compare results of different runs.
  • Create issues in an issue-tracking system automatically when a test fails.
  • Create quality dashboards for the product based on the test execution results.


 Additional Resources


If you are interested in this topic, you may find more information on what a testing framework is and on how it can be built in the following webinars on the SmartBear website:


It’s Your Time to Participate!


I described the above example not to tell you that this is the only and the right way to create a testing framework. This is just an example of what a testing framework could be. I would appreciate it if you shared your experience about how you built your testing framework and what difficulties you faced during this process. If you had some unusual tasks, it would be fascinating for everyone to learn about them and about the ways you accomplish them within your framework. Moreover, a testing framework can be something different from what I think it is, and this will make your story even more interesting for everyone.


Share your story and get a chance to get a gift from us:

  • Do you have your own testing framework?
  • How do you manage tests and test sets?
  • How do you solve the environment management part (physical PCs, VMs, cloud)?


PS: Your experience could be interesting for the TestComplete team, as well, and can have a chance to affect the future of the product. Smiley Wink


Post your comments and give Kudos for comments you like. Tomorrow, we will announce a winner of Day 2 - a community member who will get the biggest number of Kudos for his/her post. Click here to learn the Day 1 winner.

12 Days of Software Quality Schedule

Community Hero

I would suggest that, in some cases, even what tests to run can be retrieved from data.  In the framework I'm working with right now, it's a "generic" framework that is utilized for all our development projects.  So, it needs to be VERY flexible to determine what application is being tested, what environment it is on, and what test cases to run.  All that is contained in an SQL database.  By generating a set of data and then running some basic framework code, we construct the list of test cases to be run, any parameter values that need to be used for those test cases, and then report the results out to another SQL table.


Additionally, you could even have your tests be constructed as building blocks in your data.   A prototype framework that I've been working on off and on over the years uses CSV files to build tests (see a video explanation of this at https://support.smartbear.com/testcomplete/videos/smartbear-academy-testcomplete-301-training-q4/).  One file contains the list of tests to run, another file contains the steps to execute, indexed per test id.  The point of a framework like this is that you can have a bunch of coders writing the automation code for "test steps" to execute different small methods to interact with your application under test.  Then you can have non-coding type testers just add rows to CSV files to build out test runs.  They wouldn't need to know how to write code at all.... just put together the "building blocks" of test steps to construct test cases.


Suffice it to say, that what Dmitry describes above is a good set of guidelines to work through.  However, I'd like to add one more criteria to frameworks.  The framework must be built in such a way so that


  1. When the application changes (and it will), you should be able to quickly and easily make changes to your automation code to allow for those changes.  This means modularized code, shared code units, etc., so that you're not changing the same code in many different places.
  2. The staff you have can maintain the framework.  Not everyone is a coder, not everyone can build automation, not everyone is comfortable with using the automation tool.  So, take a general inventory of your staff and their skill sets and build a framework that works with them and finally
  3. Use the tool. There are a lot of different ways of doing automation out there and some good techniques that span multiple tools.  However, when you start using a tool like TestComplete, don't try necessarily to use all the same techniques you used in Selenium or UFT or whatever... while those worked well over there, TestComplete is a different tool and may have different ways of achieving the same results.  And the reverse is also true. Use the tool as the tool is presented, utilizing those features that are available (NameMapping versus building a custom POM in code, for example) and you'll get better mileage out of your framework.
Community Hero

I am in the fledgling stages of building out frameworks for all of this, so the information provided is super helpful!

Community Hero

In general, the following questions come to anxious minds when you think of having/searching/creating for test framework.

Why do I need a test framework?
What is it supposed to consists?
How does it help me, team or product / application testing?
What technology should I use?
What should one require to create a framework?

I believe, below text would address them.

Let us take users of ReadyAPI / SoapUI.

Some of the users uses the tool to call their APIs even without creating a test case which is not of much useful. I mean there is fault of them, just that requires some kind of training or guidance.

However, many users uses it very well by creating the tests by organizing them into suites.

If you look at the history and how the tool is evolved, it is created to test the SOAP services internally then, may be as framework, now evolved into Open source and Pro editions. Later transformed into Suite of Products under ReadyAPI.

So, if I take the tool as framework for a moment, now you can see the how a framework should be and how flexible / dynamic by supporting many user actions. Took this example to explain the concept of framework because most of the people use it & easy to get it. Of course, any framework with all the below features / characteristics would be embraced by the larger community.

I do not think it requires to specify any details for the software. But mentioning as customarily here.

Supporting different contracts such as WSDL, WADL and Swagger Definition, then let the user create tests with different API calls by generating sample data for the respective method / operation. If you take a look at other tools, data generation has to be done manually which conforms to underline schema for each type of request.

Simplicity & Seamless Data Flow between API calls:
Users defines the test case with multiple steps with different APIs, transferring data between steps and yet flexible with various types of supported test step types. Not to forget the power Groovy Script brings and language supported by the tool  which is very big plus. And it is very simple to design the your tests.

SetUp & TearDown:
These are very familiar buzz words for the people from Development team who write the unit tests and tests require to match certain pre & post conditions.  

This is in very much in the DNA of SmartBear products.

Power of Data-Driven Approach:
Data-Driven support in the Pro (though it is possible with Open-source by implementing on your own) to support larger sets of data which we already saw many posts about it in Yesterday's article.

Easy, yet Flexible Configurations:
Another paramount importance for any framework is to adopt the changes easily. Here mainly, taking about changes required for the different users, app server, database server details etc.,  i.e., End Point Url.

Service / interface level:
If some one wants to change the endpoint, it is possible to change the same in the project including the test request by just changing at service / interface level. Those who are not aware of this, make changes at each test step. But, over the period they learn how to use well.

From Command line arguments:
It is also possible to change the endpoint value from the command prompt itself without touching the project.

Framework should be flexible to be able to run the tests against different environments such as Development, QA, Staging, Production etc.,

Of course, in the Pro edition, it is very easy to main multiple of these.  Just choose the right environment for the tests to be executed.

But, no worry to the Open Source users regarding the same if you the property expansion and use of Properties.

Reusablity & Extensiblity:
The framework created for a product should be able to use for another product mostly with a little bit of product specific customizations & probably by writing extensions if needed.

Having said that, it should be independent of the test data.

Many of you know that the functional tests can be leveraged and do load & security testing. Also, reuse the test cases by having Run Test Case test step to call another test case.
You can implement your own test steps by building the plugins in ReadyAPI/SoapUI tools.


Prevalent Execution Modes:
Some of the users are very ambitious to use the IDEs such as Eclipse, Intellij etc to execute their tests. While it should be treated an alternative option for debugging or design the unit tests, but should not be good option only to have. Because, tests should be invoked very easily without requiring to setup project locally which is required in case of IDEs and CLI mode is better option to run the tests, IMO.

Once the tests are runnable thru CLI, with the use of default runners, or by creating custom runners and wrap them into batch file in order to execute the tests.

It would be great framework if has support of build tools such as Apache-Ant, Maven, Gradle, CI tools.

Dynamic Filtering of tests / Grouping:
Again testNG users aware of grouping of tests during the execution dynamically irrespective of suite that particular test belongs. This brings great flexibility to test & deliver things shortly and and execute required tests without compromise. At the same time, eliminate unwanted tests which can increase execution time drastically.

ReadyAPI support the same in the form of Tagging, and hope many of you are using it with smile.

Of course, we all know where to go first if something goes wrong during the tests. i.e, the importance of logging. Otherwise, user have to repeat the test manually.

Reporting & Email:
After the successful completion of tests, a good reporting makes convenient for the users including management. Usually most adapt to standard reporting such as HTML so that it would easy to maintain across the products / projects / teams.

ReadyAPI generates reports variety of formats such as csv, pdf, html etc.,
Open Source edition also can generate Junit style reports.

Who does not like to see the report? But the thing is, one has to send it. If it is to be done manually, then it takes lot of effort. If it is sent automatically by the framework at the end of the execution, Voila!

Analyzing the Test Result:
This is part of the job. But you will be lucky if the success rate is 100%. Otherwise, we all know there is work and see what is happing in the logs, report the issue etc.,

Distributed & Faster Execution:
These days in the competition world, every body wants to deliver their products / services faster than ever. In order to cope up with things, need to execute the tests more faster or distribute on multiple machines so that tests could be completed early.

Also, many institutions / corporations inclined to cloud / docker to make the software delivery in Agile.

The same can be leveraged for test execution. Why not?

How do We do?
I just wanted to briefly tell that we do use internal framework which was created based on Open Source edition (it was only available then) long back and maintained to enhance various things as per the requirements.
A template project with template cases for different test flows.

Test engineers need not require write test cases in SoapUI tool, instead Test Data is created separately i.e.,test cases are created in xml format by refering the template test cases, requests are kept in separate file and test case will have reference to that. Similarly for response comparison.

Test data is segregated as per the features into different directories and custom implementation of Tagging so that tests are filtered dynamically.

Then generate the actual project for the above test data.

Like mentioned earlier, use it with Apache-Ant tool by creating simple test targets which in turn uses testrunner of SOAPUI, generates Junit style.

Of course, any build or CI tools can be adopted without any change.

In brief:
The above mentioned points brings impeccable value to the test framework. And that is why the ReadyAPI/SoapUI got so famous amongst the API testing tools. Even if you write a test framework, or looking out for any tools, make sure the above are achievable by the tool / framework not only for APIs but for any other testing as well.

It would be more tempting to read or listen all these from various people. But you would not imagine the amount of effort went to get to that stage. You may feel if you start it to do it for your self.

In order to do develop a framework, one must have good understanding of the technologies available in the market including paid, free softwares, ability to add / implement their own requirements on top of existing libraries if any. It is completely not part time job, like some people anticipate, nothing less to a software development.

Not only required knowledge of programming language, but also various other things such creating xml schemas, databases, xsl transformations, batch scripting, externalizing of data, dealing with different formats of data (xml, csv, json) etc.. And maintain them.

Thank you for your time reading this. Hope this is useful.



Occasional Contributor

There are various kinds of framework like below 


Module Based Testing Framework
Library Architecture Testing Framework
Data Driven Testing Framework
Keyword Driven Testing Framework
Hybrid Testing Framework
Behavior Driven Development Framework
I think Hybrid is the best one where you take advantages of all different frameworks as per your project needs. You need not to stuck to a particular one
We should try to achieve below in our framework
Reusability of code
Maximum coverage
Recovery scenario
Low cost maintenance
Minimal manual intervention
Easy Reporting
Senior Member

Do you have your own testing framework?
In our case, to test the APIs we build a new framework using JUnit/TestNG and Core Java concepts.
A class is considered as a Test Suite which consists of JUnit test cases.
To call those test suites in one, Run the .class files from Main method of a single Java Class. At the end only we have to call this Main() method only.


The APIs we test have their own exceptions, to collect those and verify we use Assertions provided by JUnit.
We build a custom Exception class, which collects Assertion exception, API level exception and other(Java) exceptions for particular test case and Reset for new test cases.
Collect these different types of exception and Log it and increment values in Exception object.


After execution of a test case .log file get generate for same test.
For Test Reports we have again build a new class which manually creates .html and .txt files and write the Test Case results in those.

While writing in .html file, we write it in HTML format which ultimately shows in Grid format.
For example:
    TestID | Result | Logs | APIException | AssertException | OtherExceptions
1) TestID1 | Pass | Logs | 0 | 0 | 0
2) TestID2 | Fail | Logs | 1 | 1 | 0


How do you manage tests and test sets?
We use JUnit annotations @test for a test case and @@TestClass for a TestSuite


How do you solve the environment management part (physical PCs, VMs)?
We use VMs to certify the different environments.
To connect and run the test suite on those we use Jenkins plugins to connect to remote nodes.
We use ANT script to execute the Main() method of class which internally calls different test suites from Jenkins.
To execute it on different physical machine we need a End Point URL..

Community Manager

Hi, Thanks to all participants in the Day 2 discussion of 12 Days of Software Quality. We are ready to announce the winner. My congratulation goes to @tristaanogre! You gave Robert 7 Kudos. 


We've already posted the Day 3 article. This time, we want to talk with you about Load Testing. Are you in?


Super Contributor

I have used testing frameworks before such as JUnit TestNG or RobotFramework


Concepts are always the same regardless of testing frameworks used


Having Similar test cases in test suites

Having your set ups and tear downs.

Absolute awesome error reporting - this is something that should be an ehancement for READY Api.

There is errors but I am getting the whole request while I only care about asserts that failed.


Using CI siuch as Jenkins to drive your tests

Reporting - this is made easy with READY API JUnit report forms

you always have the option of customizing these reports by manipulating with xml reports.


This is an amazing article and we can always learn more.





w're using TC for several years now. Our testproject becomes larger and larger and one problem is the following: opening the list of available testroutines in the testitems tree is almost impossible. More than 30 seconds per new testitem! Is this normal or can routines be excluded from the list before it pops up? Many routines are not testroutines but subroutines.


Furthermore, every new TC version makes the tests a bit slower. We already did deselect all extensions that we don't need. But this is not enough though.



P. Vinders

Users online (145)
New Here?
Join us and watch the welcome video:
Join the September Hub-bub to show off, learn and win