Forum Discussion

stein_oddvar_ra's avatar
stein_oddvar_ra
Contributor
8 years ago

Parallel testing - running on two slaves in parallel

Hi all,

 

I have a config where I run one suite of test projects in one hardware setup, and a different suite of projects in another hardware setup.

I need the tests to run independently on each setup.

 

I can create jobs with one task for each setup (slave), and run the jobs. Since the tasks in a job run concurrently, this seems to solve it. But one test project  can take considerably shorter time than another, so one slave will have to wait for the other to complete before I the next job can start. Have I understood this right?

 

I have explored batch running of projects using Test Items in the project page on the master project, and this runs single projects one by one.

 

I would like to run two lists of tests on two different slave computers in parallel independent of each other.

 

Is this possible, or do I have to write a script to run the tasks one by one on the slave computers?

  • HKosova's avatar
    HKosova
    SmartBear Alumni (Retired)
    8 years ago

    Jobs runs one after another. Tasks within a job run in parallel. So you need a job with 2 tasks, where each task runs the corresponding batch of tests. As others said, this can be done by using 2 client project suites - one containing 5 tests for one client, and another one with 7 tests for the other client. In your master project, configure the tasks to run the corresponding project suites:

    • Specify the .pjs file path in Project file name (if this column is not visible, right-click and use Field Chooser).
    • Leave Test empty; this means run the entire project suite.
    • stein_oddvar_ra's avatar
      stein_oddvar_ra
      Contributor
      8 years ago

      Thanks for all your inputs.

       

      The definitions of suite, project and test/testcase was useful. I find that I already use suites and projects, but in my setup, the projects contain one test/testcase each.

       

      In other words, I have one server that runs a suite containing the client tests projects for two clients as a list of jobs.

       

      Creating a master suite to run the two client suites should be easy. Creating a client suite with all the projects/tests seems to be a bit less straigthforward. I add projects/tests to the project page from already defined jobs, and now we have one master running a suite that starts a master suite on a client, that in turn starts slave projects/testcases on itself (the same client).

       

      Now, I find this to actually work - after a fashion.

       

      If I put the host name (in hosts in TestComplete) in the host address, I get an error when I try to run the project:  "The local computer cannot be used as a slave host."

       

      If I instead use the ip-address of the local machine I can actually run the project, and the testcases (sub-projects or slave projects defined in jobs) run one by one. And I have two indicators in the top right of the screen (see attachement).

       

      Is this the way to actually make this work? Or am I doing something that potentially could give me problems?

      • tristaanogre's avatar
        tristaanogre
        Esteemed Contributor
        8 years ago

        This could potentially give you problems.  The idea of a Network suite is documented that you have multiple machines in order to distribute the testing to give a better throughput by sharing out the work.  Having everything run on the same machine just doesn't make sense.

        But even further than that... if project 1 wants to use the mouse to click on something and project 2 wants to use the mouse to click on something different, which one wins?

        Also...  I would suggest a re-architecture...  Having only one test case per project... well, if your application continues to grow, that will mean you could have 30, 40, 100, 200, etc. projects...  it's not a sustainable architecture. 

  • m_essaid's avatar
    m_essaid
    Valued Contributor
    8 years ago

    hi,

    in additino to what tristaanogre said, if you want to start 2 project suites in parrallel I assume you have to licences of TestExecute.

    a tip : you can launch the 2 projects suites but you won't be able to work on them simultaneously with TestComplete.

    So for night runs, it could be the same projects suite, but for the day, make copies and use VM such as HyperV on your local machine...

    • tristaanogre's avatar
      tristaanogre
      Esteemed Contributor
      8 years ago

      Yup, what m_essaid said...

       

      You can achieve basically the same effect using a source control tool like Git or Mercurial.  Check out the solution presented here. In your case, you would have your dev machine and your two running environment machines.  As mentioned, this requires licenses... most likely a single license of TestComplete and a couple of TestExecute licenses.

  • tristaanogre's avatar
    tristaanogre
    Esteemed Contributor
    8 years ago

    I guess I'm not understanding the question.  If I'm reading it correctly, you want to be able to run two different project suites on two different machines.

    So... architecturally, in my mind, I don't see the need for setting up a Network suite in TestComplete.  Basically, you create two different PJS files (Project Suites), one for each environment with the necessary projects associated.  Then, using some kind of scheduling tool (Windows Schedular, Jenkins, etc), fire off the project suites on each machine.  You should be able to do this without needing to wait for one to finish before you kick off the other.  

  • stein_oddvar_ra's avatar
    stein_oddvar_ra
    Contributor
    8 years ago

    I'm sorry if I'm unclear, I'll try to explain more:

     

    I have a suite of separate tests (12) to run on two client/slave computers. 5 on one, and 7 on the other. One server has a master pjs-file, and controls the test run on the client/slave computers.

     

    I can setup a job with two tasks to run one sub-tests on two clients in parallel, but the job finishes only when the last client has finished and the other client is idle until then. I would like to run five tests on one client and seven on the other. The total time each client runs is fairly equal, but the individual tests clocks in differently.

     

    I can also setup batch running in the project page to run all slave projects in succession, but then one client is idle, while the other runs a test.

     

    In the documentation for TestComplete there is one page describing parallel testing the way I would like it to work: https://support.smartbear.com/testcomplete/docs/testing-approaches/parallel-testing.html

     

    The setup I have corresponds to this description, except that I have more than one test to run on each slave computer; one after the other.

     

    The only way I have found to do this is to make a controller script loop on the master computer that starts tests and checks the status on the slaves.

     

    Is this the only option I have in my case?

     

    PS: I have the licenses I need. TestComplete on the servers, and TestExecute on the client/slave machines, so that is not an issue.

    • tristaanogre's avatar
      tristaanogre
      Esteemed Contributor
      8 years ago
      stein_oddvar_ra wrote:

      The setup I have corresponds to this description, except that I have more than one test to run on each slave computer; one after the other.

       

      The only way I have found to do this is to make a controller script loop on the master computer that starts tests and checks the status on the slaves.

       

      So... you have multiple tests to run on the slaves.  Let's clarifiy some definitions...

      PJS = Project Suite.  This is an organization in TestComplete that has one or more projects.  The PJS itself can be run to execute those projects in a configured order

      Project = A collection of tests to be executed, stored in an MDS file. Usually, a project will contain multiple test cases, either as Keyword tests, script routines, or some sort of framework structure for executing tests.

      Test = A specific test case that executes to verify/validate a portion of the AUT.

      These are the definitions I operate from.  so, you say you have 12 tests to run on on two client/slave computers.  Are you using "test" in the same way I am? Or are your "tests" actually projects?  The answer actually matters because the structure indicated in that article you posted assumes that each item running in parallel is a project that contains multiple tests.  So, you can execute, on two slave machines, two projects in parallel.  

      If your 12 "tests" are 12 "projects" that you want to just split up among the two machines, your best bet is what has already been suggested... organize your projects into two project suites, one with 5 the other with 7, and use some external tool to kick off those project suites at the same time.

      Now, in looking at that article, it does seem to imply that a master project can execute project SUITES on the two slave machines... HKosova might be able to comment on that... if that's the case, the organization I mentioned should still work just that you would use your Master to kick of the suites, not the individual projects.

    • m_essaid's avatar
      m_essaid
      Valued Contributor
      8 years ago

      hi stein,

      forget parallel testing, it's a real pain, do it yourself.

      put your 7 tests in a project, your 5 other in another project.

      use robocopy or the tool you want to make two copies of the whole project suite.

      then run simultaneously the 2 TestExecute