Ask a Question

Running tests overnight using separate capabilities

SOLVED
jaredjamieson
Occasional Contributor

Running tests overnight using separate capabilities

Hi all,

 

My job is to test a mobile device application using a long list of 23 different iPhones and 3 different iOS settings.  I want to setup a sequential test plan where one phone will be run across each iOS version, then move on to the next phone, rinse and repeat.  I see there is an option to do this using parallel device cloud environments but running 23x3 simulated devices all at once sounds like a task my laptop cannot handle.  

 

Is this use-case possible in TestComplete?  Is there an easy way to task runtime environments to an execution plan without coupling that test case to each individual environment?  I.E. Test Case "launch app" can be run using the overnight test case runner but I can also select just that test case to run on a connected device only.  

 

My setup is this: TestComplete v. 15.44.11x64 is linked to a Windows 10 laptop, talking to Mac Venture 13.0 Laptop with Appium 1.22.3 and XCode 14.1 running on the Mac.  

 

Feel free to offer any advice possible.  This is my first time using TestComplete (its been about a month now), so I appreciate any feedback!

3 REPLIES 3
Marsha_R
Community Hero

@AlexKaras have you done anything like this?


Marsha_R
[Community Hero]
____
[Community Heroes] are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Posts made by [Community Heroes]
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
The [Community Hero] signature is used with permission by SmartBear Software.
https://community.smartbear.com/t5/custom/page/page-id/hall-of-fame
AlexKaras
Community Hero

Hi,

 

This is my first time using TestComplete

TestComplete provides extremely great documentation, so I would recommend to get a habit to refer to documentation first in case of any question. It really worth detailed reading, especially for novices.

 

Back to your question and I hope that I got it right...

Are you using local device farm or cloud one like Bitbar/BrowserStack?

I just skimmed the relevant parts of TC documentation (sorry) and had not have to implement configuration like yours yet, so my rather theoretical opinion is like this:

For the local device farm you might indeed expect lack of resources from your laptop (I guess it will be Mac one). For the cloud-based farm, most probably you will have a restriction with the number of licensed parallel test executions.

Considering the above, my current idea is to consider some helper test configuration file and some helper driver code.

Configuration file might be like this:
{

  devices : {

    1 : "iPhone 6",

    2 : "iPhone 7",

    ...

  },

  iOSconfigs : {

    1 : {config1},

    2 : {config2},

    ...

  }

}

And the driver code will iterate them like

for (each device in devices) {

  for (each iOSconfig in iOSconfigs) {

    <driver code>

  }

}

And the <driver code> will craft required set of capabilities and start test execution via Parallels object provided by TC. To control the number of parallel executions you may either implement some logic inside <driver code> or try to use the Parallel Test Count option in TestComplete.

 

Another option that may be considered is the use of some CI/CD server.

In this case you may create either 23x3 test wrappers or 23x3 test configuration files with one test wrapper that will get required configuration as a parameter. Then you will setup a job on your CI/CD server and populate this job with 23x3 steps. Each step must be configured as running in parallel so that CI/CD server does not run them sequentially. Then you will have to configure the agent(s) that are used to execute these steps to execute not more than certain number of tasks (say 5) simultaneously. As a result, 5 tests will be started and all other will be queued. And than it will be the task for the agent to handle the queue until all queued steps are not completed.

 

Documentation articles:

https://support.smartbear.com/testcomplete/docs/working-with/managing-projects/execution-plan/enviro...

https://support.smartbear.com/testcomplete/docs/testing-with/running/parallel.html

https://support.smartbear.com/testcomplete/docs/reference/options/engines/parallel.html

 

P.S. @Marsha_R : Thank you a lot for pinging me 🙂

 

Regards,
  /Alex [Community Hero]
____
[Community Heroes] are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Posts made by [Community Heroes]
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
The [Community Hero] signature is used with permission by SmartBear Software.
https://community.smartbear.com/t5/custom/page/page-id/hall-of-fame
================================
jaredjamieson
Occasional Contributor

Thank you Alex.  I'll give this a try when my work brings me into setting up overnight tests using the various iOS and iPhone tests! And thank you for pointing me at several possible options for completing such a task.  

 

To answer your question, I am using a local Appium server.  I am not sure what you refer to as a device farm.  I select a specific device using JSON like as shown:

{"platformName":"iOS", "platformVersion":"15.5", "deviceName":"iPhone 8", "automationName":"XCUITest", "updatedWDABundleId":"my.example.app"}

As well as I specify the server URL and path to the app build (*.app.zip).

 

I have been spending lots of time reading documentation on TestComplete's website however I have had no luck finding a strategy for fixing my problem.  I am looking for a way to 'bind' all the phones and iOS instances to the TestComplete user and to then apply those to one or more tests I select under the Execution plan.  The way I was implementing it was to go to Execution Plan -> Select specific test case -> Parallel Device Cloud Environment -> Local Appium -> New environment (or import from file) -> have 23x3 instances of environments to handle every phone with every iOS.  This fails due to a lack of resources on the server and will only run the test on every environment instead of selecting one or more environments.  I want to be able to run a test using the directly connected device (one phone on one iOS) or to select specific phones and iOS builds.  I figure this exact scenario is not possible in TestComplete.  Running tests through the parallel cloud environments did not feel like an optimal solution so I turned to this community post.  

 

A question leading off of your solution I have is: is running a test case on multiple iOS builds and phones not a common task for test complete?  How do other professionals using TestComplete fulfil a necessary testing framework?  Every singe Android and Apple application has to go through some sort of testing on all phone devices and build versions.  

 

Thank you for your time. 🙂

cancel
Showing results for 
Search instead for 
Did you mean: