Wintertainment 2020 - Day 1: Increasing Test Coverage and Scalability With the Device Cloud



Wintertainment 2020 is here! Woo-hoo!🎉


We start off with this awesome article written by a great TestComplete expert @hkim5. The article speaks about test coverage and scalability - these are very important topics in modern testing.


Read the article and answer the questions at the end of it to share your experience and get a chance to win prizes from SmartBear! Read participation rules and details by this link

Let's rock!


Increasing Test Coverage and Scalability With the Device Cloud


In QA, the expectation used to be that there would be a price to pay for accelerating our software development lifecycle, whether that be quality at the expense of speed, or speed at the expense of quality. As teams become more agile to meet the velocity of a growing industry, we still have the goal to deliver good, bug-free software – just now at pace! 


Introducing test automation to your QA process is the first step, but how do we go even faster if we already have automation in place? The logical next step would be to accelerate this even further by implementing parallel testing. Instead of running one test sequentially after another, parallel testing would distribute the testing load across multiple nodes and run multiple tests at the same time.  


Time for some quick math:  


Imagine your team has 50 automated tests and each test takes about two minutes to run, totaling 100 minutes of execution time. Now imagine you had 5 parallel licenses – running these tests concurrently would mean the test execution time goes from 100 minutes to 20 minutes. That is an easily quantifiable ROI, and more time that goes back to your QA or dev teams.


Don’t Forget the Importance of Scalability


I’m a strong proponent of the idea that not 100% of tests can be automated. Plus, nothing could possibly replace exploratory testing anyways, since using your background knowledge and intuition to find end user production issues is an invaluable art of the QA process. By running parallel tests and saving time on our most repetitive and time-consuming tasks, we have more time to focus on the details (or talking new features, work on future releases, etc.) 


Since the release of TestComplete 14.4, we’ve revamped our integration with our device cloud. The functional tests that we’ve built for our web app can now be deployed on a remote browser, where we can leverage the 2000+ different combinations of operating systems, browsers, versions, and resolution settings. This means we can test previously inaccessible MAC or Linux based browsers.  


Furthermore, relying on the  SmartBear remote device cloud also means we can achieve parallel testing without needing to rely on the provisioning of an internal Selenium grid (the total cost of ownership and management costs alone would be daunting for many sys admins). You can either hard-code the remote browser capabilities into your test scripts, or refactor it such that we can define the capabilities at runtime with a custom command line argument. 




Fully functional sample code snippets can be found here within our documentation 



Whether these capabilities are hard-coded or not, this affords us a huge amount of flexibility in terms of deployment by enabling us to test our web application throughout the various browser and OS configurations that we need to validate prior to release.  

Less Maintenance, Expanded Re-Use 


Imagine if we had a valid set of smoke tests or a regression suite that we needed to run on tens, if not hundreds of browser configurations. With the Device CloudTestComplete can verify each variation of the test run on each of the configurations using a single test (set). After all, this type of reusability and low maintenance is what makes for a good automation framework/test.  


These tests can be launched in parallel using the test runner applicationTestComplete has native support for this test runner in both Jenkins and Azure DevOps (your CI/CD administrator would be able to install the free plugin for you). For each of the pipeline builds targeting however many parallels, you will receive the familiar reports just like before. The only difference is that we have now expanded our test coverage to include previously unavailable browser/OS configurations, as well as having cut down on our execution time by 1/#of  parallels (the quick math section above). 


Expand Testing Beyond Parallel 

Starting with TestComplete 14.7, we’ve introduced a Parallel scripting object, where users can now trigger parallel executions leveraging a remote device cloud without a CI/CD framework. So, for whatever reasons, if we don’t yet have a CI/CD framework or an agent node provisioned to us yet, we can leverage the parallel executions from within TestComplete by using the new scripting object.  


Scaling your TestComplete tests to the cloud across different platforms, devices, and configurations was already a great way to accelerate your testing and increase test coverage. Now you’ll have the ability to run these tests in parallel in the cloud – saving time and infrastructure costs – and introducing incredible scale to your testing process.


Further Reading: 



What do you think? Comment below! 


1. How are you expanding your web application test coverage throughout the various browsers and OS configurations that need to be tested? 

2. How are you currently reusing/maintaining automation scripts? Especially throughout future releases of your web application? 

Community Manager
Senior Member
  1. For various browsers and OS testing, we need to consider the below combinations and run the same test case so many times.

Browser * devices * OS * browser versions

Solution: Parallel Testing via cross-browser automation tool, where multiple environments are tested in parallel. And hence the testing is completed in less time.

Different screen resolutions may break the page layout

When rendered on devices of different resolutions, the page layout is likely to break. Manual testing on all devices with different resolutions is difficult to achieve.

Solution: An automation tool which easily lets you automate your test cases and then execute them in parallel on unlimited devices of different resolutions, in parallel to save on your precious time.

Frequent updates to browsers

Browsers receive frequent updates and different versions are released quickly. To cover every newly released version, the web application should be tested again.

Solution: Use a cloud-based cross-browser testing tool where we can just add the new browser version and the testing will take place automatically.

  1. Taking below steps to maintain automation scripts:

•Steady State Runs

•Test Run Reporting

•Script Maintenance

•Framework upgradation and Maintenance

Community Manager

@mandhale Thank you for your detailed answer! I completely agree, cross-browser automation tools are making lives of testers so much easier. It is minutes versus hours of work!

Nice comment. 

Frequent Visitor

I'm a new user of That's what we plan to use to expand our testing across desktop and mobile. It allows me to select up to 25 desktop/mobile browser combinations to do a screenshot comparison and indicates when there are layout differences.  Unfortunately we do not have automation in place but I'm working on using the Record and Replay feature of crossbrowsertesting to help reduce the manual test time on our small projects. 

Community Hero

Where was Device Cloud when I needed it at my last job?  🙂    Our TC tests were quite extensive there and could take hours to run.  This would have been very handy.


At my current job, our test set is still pretty small and they run fairly fast, but I will definitely plan for this in the future.  


About Device Cloud

I had focused on the Device Cloud function. It looks very convenient to cross test.

Our automation tests were still small in web. I decided to try it for prepare in the future. 

I have some confusing points about Device Cloud.

1.Does the Device Cloud include the NetworkSuit function?

2.If our test server is installed in local pc(we visit it by LAN ip), could I use Device Cloud to test it?


About test or maintain

We run the steady test cases at night to reduce the cost.

I think the unified regular how/which web elements should be mapped is the most import.

We could maintain test cases quickly if we have the rule.


Hi @ApplePen , 


In answer to your questions;


  1. Device Cloud is independent of Network Suite. However, you can still run Device Cloud tests on hosts you manage through Network Suite if you choose. 

  2. Yes, you can use a "local connection" - an encrypted tunnel between your computer and the CrossBrowserTesting device cloud. The CrossBrowserTesting service will route all traffic to your tested web application through your computer.


To learn more about local tunnels and how to open them, see the following documentation:

Community Hero

Points which i take into consideration while coverage:


1. Frist, Prioritizing the Devices and Browser which we are considering for Testing by setting up a meeting with Team and Client.

2. Keep most popular device and browser into consideration like Xiaomi, Samsung, Motorola etc. while calculating the Test Case coverage.

3. Third is, OS Popularity and it's adoption we need to take this into consideration for different OS(s) available in market like Android, iOS, Symbian, etc.


How are you currently reusing/maintaining automation scripts? Especially throughout future releases of your web application


For this what we are doing is Execution weekly and if our scripts failed then we are looking into result if it is failing because of change then we are updating the Script if not then we are raising defect for the same


If anyone is having points to consider please write out 🙂


Occasional Contributor

Certainly a thought-provoking article. I have no answers to the questions at this time, but the information presented might become valuable in the future.

Users online (337)
New Here?
Join us and watch the welcome video: