Forum Discussion

BenjaminTenne's avatar
BenjaminTenne
New Contributor
13 years ago

Odd behaviour of 'step average' load test assertions

Hi,

I've been having difficult with 'step average' assertions on my load tests (soapUI 4.5.1). I've found no reference to such problems on the forum, so I'm prepared to have it pointed out that I've misunderstood something and/or I'm an idiot

My load test is very simple. I set a 'total runs' limit of 100, set threads=1, test delay=0, random=0. In other words, I'm running 100 iterations of a test case, one at a time, each starting immediately after the previous completes.

Let's say my test has a single step called 'Step A'. I've observed that this typically takes 200ms (i.e. '200' appears in the avg) column and want to add a 'step average' assertion to ensure it never exceeds 300ms. I want this assertion to be applied once, as the end of the test run, so I set the parameters as follows:

minimum request: 100 (i.e. don't evaluate the assertion until after all iterations have completed)
max average: 10 (I'm setting it way lower than the 200 I've observed, as I want to deliberately trigger the assertion every time; I'd change it to 300 once I'm happy)
max errors: 0
sample interval: 100

I run this and the assertion isn't triggered. I change 'sample interval' to 50, and that makes no difference. However, after various experimentation, I sometimes end up in the state where the assertion fires after 2 or 3 iterations, seemingly regardless of how I've set these parameters.

Am I misunderstanding something? Can someone tell me what parameters I should set to have the assertion applied once, at the end of my 100 iteration run?

Many thanks,
Ben.

1 Reply

  • BenjaminTenne's avatar
    BenjaminTenne
    New Contributor
    13 years ago
    I realised that the previous example I gave is flawed. Setting the max average to 10ms means that test can reasonably fail once the total elapsed time exceeds 1000ms (i.e. 100 x 10ms), given that it can't possibly succeed, even if the remaining iterations take no time at all. However, it was one of many attempts I'd made, most of which were more sensible.

    So, here's a concrete example of something I've just tried.

    I have a test step called Create Order, with a Step Average assertion, as follows:

    Minimum requests: 80
    Max average: 500
    Max errors: 0
    Sample interval: 20

    My load test is set to Threads:1, Strategy: Simple, Test Delay:0, Random: 0.0, Limit: 100 total runs.

    It failed on my assertion after two iterations:

    Average [760] exceeds limit [500] [threadIndex=0]

    The stats grid for the Create Order step shows:

    Min: 372, Max: 760, Avg: 566, Last: 372, Cnt: 2

    So, it's done two iterations, taking 372 and 760ms, giving an average of 566ms. But why did the assertion fire before the 80 minimum requests was reached? The tooltip for Minimum Requests shows "Minimum number of steps before asserting". Even if that meant total steps (rather than the step I'm asserting on), my test case only has 14 steps, so 2x14 is nowhere near 80.

    I'd be very grateful if someone can point out what I'm doing wrong, as this is making the whole approach unworkable at the moment.

    Thanks,
    Ben.