Forum Discussion

ricer333's avatar
ricer333
Occasional Contributor
9 years ago

Load Tests are not fully executing

Spoiler
Test Step,min,max,avg,last,cnt,tps,bytes,bps,err,rat
getSubjectEntitlements for user STEST21,55,399,198.18,133,50,4.02,9200,741,24,48
getSubjectEntitlements for user STEST24,62,334,177.5,169,26,2.09,4784,385,9,34
getSubjectEntitlements for user STEST26,179,680,448.23,541,17,1.36,10455,842,15,88
TestCase:,296,1413,442.88,843,50,4.02,24439,1969,48,96
~

This is output from a LoadTest that I just ran. You can see that the cnt is different per TestStep 1 vs 2 vs 3

50 for test step 1,

26 for test step 2,

17 for test step 3

 

I have designed this particular test as a Simple Strategy with the following params:

Threads: 5,  Test Delay: 1000, Random: 0.5

Limit: 10, Runs per Thread

 

There are assertions on these tests, HTTP Valid Heades, SLA, and JSON Script verification

 

I would expect each cnt to be 50. Am I missing someting? Why would test steps 2 & 3 fall short by 50% and almost 60%?

 

 

  • ricer333's avatar
    ricer333
    Occasional Contributor
    9 years ago

    Maybe I mis-interpretted how the 'Runs per Thread' works? I assumed that each test step would be given 5 threads and each test step would run 10x per thread; equally 50 executions.

     

    Since the 3rd test step is taking longer, it is only executing around 17-20 times. It looks like the LoadTest ends as soon as any one of the test steps reaches the 50 limit.

     

    Is there any way to write the tests so that each of these runs a total of 50x? 

     

    Should each test step be it's own test case instead, with its own LoadTest associated with it?

    • rupert_anderson's avatar
      rupert_anderson
      Valued Contributor
      9 years ago

      Hi,

       

      Its hard guess exactly what might be wrong without seing your TestCase details. In my experience "runs per thread" should mean exactly what you expected e.g. 5 threads, limit 10, "runs per thread", should result in the TestCase being run a total of 50 times, 10 times per thread. However, there are situations where things can get somewhat unpredictable! This can be because there is some contention between the threads when they all execute your TestCase simultaneously - for example, say the TestCase makes use of some "shared" resource (e.g. property, datasource), then there can be concurrent updates by the threads, although in your case, I think the delay makes this type of contention less likely, just a guess.

       

      Have you tried running your TestCase with 1 thread, do you get the expected number of runs?

       

      Regards,

      Rupert

      • ricer333's avatar
        ricer333
        Occasional Contributor
        9 years ago

        I set the Runs per Thread to 1 and executed with 5 total threads.

         

        When teststep1 hit 5, the testcase stopped.

         

        If I want to test 3 different users hitting the same REST API, should these be separate test cases within a test suite, each with their own load test associated with them?

         

        Or should it be as I have it now, which is: three separate teststeps, in 1 test case with 1 loadtest testing all three 'at the same time'?

         

        What is the better practice here?