Ask a Question

Wintertainment 2020 - Day 1: Increasing Test Coverage and Scalability With the Device Cloud



Wintertainment 2020 is here! Woo-hoo!🎉


We start off with this awesome article written by a great TestComplete expert @hkim5. The article speaks about test coverage and scalability - these are very important topics in modern testing.


Read the article and answer the questions at the end of it to share your experience and get a chance to win prizes from SmartBear! Read participation rules and details by this link

Let's rock!


Increasing Test Coverage and Scalability With the Device Cloud


In QA, the expectation used to be that there would be a price to pay for accelerating our software development lifecycle, whether that be quality at the expense of speed, or speed at the expense of quality. As teams become more agile to meet the velocity of a growing industry, we still have the goal to deliver good, bug-free software – just now at pace! 


Introducing test automation to your QA process is the first step, but how do we go even faster if we already have automation in place? The logical next step would be to accelerate this even further by implementing parallel testing. Instead of running one test sequentially after another, parallel testing would distribute the testing load across multiple nodes and run multiple tests at the same time.  


Time for some quick math:  


Imagine your team has 50 automated tests and each test takes about two minutes to run, totaling 100 minutes of execution time. Now imagine you had 5 parallel licenses – running these tests concurrently would mean the test execution time goes from 100 minutes to 20 minutes. That is an easily quantifiable ROI, and more time that goes back to your QA or dev teams.


Don’t Forget the Importance of Scalability


I’m a strong proponent of the idea that not 100% of tests can be automated. Plus, nothing could possibly replace exploratory testing anyways, since using your background knowledge and intuition to find end user production issues is an invaluable art of the QA process. By running parallel tests and saving time on our most repetitive and time-consuming tasks, we have more time to focus on the details (or talking new features, work on future releases, etc.) 


Since the release of TestComplete 14.4, we’ve revamped our integration with our device cloud. The functional tests that we’ve built for our web app can now be deployed on a remote browser, where we can leverage the 2000+ different combinations of operating systems, browsers, versions, and resolution settings. This means we can test previously inaccessible MAC or Linux based browsers.  


Furthermore, relying on the  SmartBear remote device cloud also means we can achieve parallel testing without needing to rely on the provisioning of an internal Selenium grid (the total cost of ownership and management costs alone would be daunting for many sys admins). You can either hard-code the remote browser capabilities into your test scripts, or refactor it such that we can define the capabilities at runtime with a custom command line argument. 




Fully functional sample code snippets can be found here within our documentation 



Whether these capabilities are hard-coded or not, this affords us a huge amount of flexibility in terms of deployment by enabling us to test our web application throughout the various browser and OS configurations that we need to validate prior to release.  

Less Maintenance, Expanded Re-Use 


Imagine if we had a valid set of smoke tests or a regression suite that we needed to run on tens, if not hundreds of browser configurations. With the Device CloudTestComplete can verify each variation of the test run on each of the configurations using a single test (set). After all, this type of reusability and low maintenance is what makes for a good automation framework/test.  


These tests can be launched in parallel using the test runner applicationTestComplete has native support for this test runner in both Jenkins and Azure DevOps (your CI/CD administrator would be able to install the free plugin for you). For each of the pipeline builds targeting however many parallels, you will receive the familiar reports just like before. The only difference is that we have now expanded our test coverage to include previously unavailable browser/OS configurations, as well as having cut down on our execution time by 1/#of  parallels (the quick math section above). 


Expand Testing Beyond Parallel 

Starting with TestComplete 14.7, we’ve introduced a Parallel scripting object, where users can now trigger parallel executions leveraging a remote device cloud without a CI/CD framework. So, for whatever reasons, if we don’t yet have a CI/CD framework or an agent node provisioned to us yet, we can leverage the parallel executions from within TestComplete by using the new scripting object.  


Scaling your TestComplete tests to the cloud across different platforms, devices, and configurations was already a great way to accelerate your testing and increase test coverage. Now you’ll have the ability to run these tests in parallel in the cloud – saving time and infrastructure costs – and introducing incredible scale to your testing process.


Further Reading: 



What do you think? Comment below! 


1. How are you expanding your web application test coverage throughout the various browsers and OS configurations that need to be tested? 

2. How are you currently reusing/maintaining automation scripts? Especially throughout future releases of your web application? 

SmartBear Alumni (Retired)