Forum Discussion

dm's avatar
dm
Occasional Contributor
5 years ago

Is there a way to reduce how many tests are run in parallel, via testrunner.bat

Context: we currently run our project in 'PARALLELL' mode, with most suites set to 'SEQUENTIAL'. This works fine on our Jenkins box, running against physical application servers. We're moving to the cloud in general, and currently hoping to spec the servers there to be smaller than their equivalent real-world servers. When we have been running with the project set to SEQUENTIAL, we can get the tests to pass, but they obviously take a long time. When we run in PARALLELL we can only get them to complete at all if we set the application server to be as large as their real-world equivalent.

 

My problem: I assumed threadpool.max being set in testrunner.bat would help reduce how much we hit the application server. Each time I reduce it to a threadpool.max value that I think the server could survive on, the testrunner process completely hangs and never recovers.

 

The questions:

  1. 1) is there a way to run in 'PARALLELL' mode at a project level, but restrict how many tests are run at any one time? eg: say 'PARALLELL' but only run 4 tests simultaneously
  2. 2) what does threadpool.max do? Is there other commands I can add to testrunner.bat that modify the behaviour how I want it to? I basically want to say 'run as parallel as you can but only run 4 tests simultaneously'
  3. 3) If I set the project to run 'SEQUENTIAL' what does that mean? Does it mean 'run each test suite in sequence and then honour their settings'?
  4. 4) Can we get PARALLELL renamed to PARALLEL?

 

Thanks in advance for any help

  • nmrao's avatar
    nmrao
    5 years ago
    If 1700 tests in single project, then you can spilt into 4(as you wanted to run max 4 parallel) projects and invoke all of them. This is naive and simple if you can.
  • richie's avatar
    richie
    Community Hero
    5 years ago
    Hey dm,

    No one else has responded to this as yet .... i cant answer all your questions but i might be able to help on a couple of them

    Answer Q1. In regards to controlling the number of tests that are executed, can you not use tagging? There's a testRunner switch you can use to use the tagging i believe. If you google "soapui testrunner tagging" and hit the first link returned this mentions the use of tags via testRunner execution.

    Answer Q3. In regards to the difference between sequential vs parallel i can only comment on the behaviour ive seen. Sequential executes each testsuite within a project sequentially, all tests within a testsuite needs to execute and complete before moving to the next test. Once all tests in testsuite have executed, the execution of the next testsuite begins.
    I don't really ever run in parallel mode so i cant comment on that im afraid.

    Q4. Smartbear are probably aware of "parallel" being incorrectly spelled. If this was the licensed Pro (ReadyAPI!) version id suggest submitting a ticket with Smartbear. As SoapUI is open source i dont really know how bug fixing actually works.

    Cheers,

    Rich
    • dm's avatar
      dm
      Occasional Contributor
      5 years ago

      Thanks Richie

      Re 1 - I want to run all the enabled tests (~1,700) in the project, so tags isn't the solution for me here (unless I ran in sequential groups of tags over many runs I guess)


      Re 3 - thanks, that's useful to have confirmed. Sequential at the project level seems to mean 'sequentially run each suite'. I'm unclear if the suite's settings then take over (eg: run Suite #1 in parallel, and Suite #2 in sequence, and Suite #3 in parallel)?

      I guess.. given the answer to #3 here... my question changes a bit: is there a way to run less suites in Parallel than what appears to be 'all' of them? eg: a way to run only 2 suites concurrently, instead of say... 50+

      • richie's avatar
        richie
        Community Hero
        5 years ago
        Hey dm,

        In regards to point 3, i have run tests in parallel before now, just not often so don't really feel qualified to give much of an opinion.

        Im not in front of my laptop at moment (i will be later tonight) so i can double check, but i do believe running in parallel has some limits. If im remembering correctly when ive had a large number of testsuite in my project and ran parallel it didnt execute every single testsuite simultaneously. I think it hit some sort of resource or setting or simultaneous connection limit and once that was achieved the execution cycle waited for thr execution of a testsuite was completed before moving onto the next.
        Have a look in your http settings. Is there a max connections to host setting or something equivalent?

        Someone asked this very question (about limiting n threads of execution in parallel run) on the forum in 2013 and Smartbear support said it wasnt possible in soapui but was possible in the loadui component. Who knows? Things may have changed in the last 7 years....

        Thats all i got im afraid!

        Nice one

        Rich