Forum Discussion

dm's avatar
dm
Occasional Contributor
5 years ago

Is there a way to reduce how many tests are run in parallel, via testrunner.bat

Context: we currently run our project in 'PARALLELL' mode, with most suites set to 'SEQUENTIAL'. This works fine on our Jenkins box, running against physical application servers. We're moving to the ...
  • nmrao's avatar
    nmrao
    5 years ago
    If 1700 tests in single project, then you can spilt into 4(as you wanted to run max 4 parallel) projects and invoke all of them. This is naive and simple if you can.
dm's avatar
dm
Occasional Contributor

Thanks Richie

Re 1 - I want to run all the enabled tests (~1,700) in the project, so tags isn't the solution for me here (unless I ran in sequential groups of tags over many runs I guess)


Re 3 - thanks, that's useful to have confirmed. Sequential at the project level seems to mean 'sequentially run each suite'. I'm unclear if the suite's settings then take over (eg: run Suite #1 in parallel, and Suite #2 in sequence, and Suite #3 in parallel)?

I guess.. given the answer to #3 here... my question changes a bit: is there a way to run less suites in Parallel than what appears to be 'all' of them? eg: a way to run only 2 suites concurrently, instead of say... 50+

richie's avatar
richie
Community Hero
5 years ago
Hey dm,

In regards to point 3, i have run tests in parallel before now, just not often so don't really feel qualified to give much of an opinion.

Im not in front of my laptop at moment (i will be later tonight) so i can double check, but i do believe running in parallel has some limits. If im remembering correctly when ive had a large number of testsuite in my project and ran parallel it didnt execute every single testsuite simultaneously. I think it hit some sort of resource or setting or simultaneous connection limit and once that was achieved the execution cycle waited for thr execution of a testsuite was completed before moving onto the next.
Have a look in your http settings. Is there a max connections to host setting or something equivalent?

Someone asked this very question (about limiting n threads of execution in parallel run) on the forum in 2013 and Smartbear support said it wasnt possible in soapui but was possible in the loadui component. Who knows? Things may have changed in the last 7 years....

Thats all i got im afraid!

Nice one

Rich
  • dm's avatar
    dm
    Occasional Contributor
    5 years ago

    Hi Richie

    Thanks for your help, yeah with the number of test suites we have the UI actually just crashes if we try to run at the Project level 'in parallel'. We have spent a couple days trying to get max HTTP thread counts to be restricted so we only run one test suite at a time, but have been unsuccessful in applying this to testrunner.bat without everything crashing.

     

    Might just have to leave this as unresolved or resolve as 'feature request'

     

    Thanks for your help

    • richie's avatar
      richie
      Community Hero
      5 years ago
      Hey dm,

      Where you state "so we only run one test suite at a time"

      Are you saying youre struggling to run your testsuites sequentially via testrunner? Cos thats easy to do...just ensure its not set on parallel before execution...or have i misunderstood again?

      Nice one

      Rich
      • dm's avatar
        dm
        Occasional Contributor
        5 years ago

        Apologies Richie, I mistyped there.

         

        My settled 'ask' (and I think you're right, I need to raise this as a support ticket with Ready API) is: can we be provided a way to set:
        - project to run 'Parallel'
        - suites to run 'Sequential'
        - but restrict how many suites are run at any one time
        eg: I would like to have 60 test suites run in parallel, but only run 3 test suites simultaneously

         

        It doesn't appear possible to use the HTTP maxthreadcount variable to manipulate this. If there's a way to achieve this goal, it's unclear to me.

         

        If I wanted to restrict parallelism to 3 tests maximum at any one time, it appears my options are:
        - restructure the project of ~60 test suites to be ~3 test suites with lots and lots of tests
        - restructure the tests into 3 projects of 1 test suite each, and run those 3 projects in 3 testrunner.bat processes
        - pay for a bigger cloud-based environment that can handle ~60 tests from ~60 test suites all kicking off simultaneously
        - advise against ever running 'all the tests' and instead get testers to run sequentially for subsets of the tests, using Tags as a way of creating those subsets

        This is a serious issue for our longterm usage of the Ready API product. Without a way to restrict parallelism, we basically are being asked to restructure our approach to test management, or come up with execution-time workarounds to avoid DOSing out environments.

        But yeah, sounds like a support case is the better way forwards from here