Forum Discussion

PeterKlim's avatar
PeterKlim
Contributor
11 years ago

Simple vs Fixed Strategy

I've read the difference at the website but it is very confusing for me and my team.

I have a load test where the test case has 40 threads (users) sharing 400 XMLslocated in one directory. Each XML is used to submit a request. I have Limit = 1 Run Per Thread.

Here is what I THINK the Strategy does.

SIMPLE:
When starting the Load test run, each of the 40 users start at the same time. Each does 1 request. Only after all 40 request return a response will each of the 40 users submits another respononse - this gets repeated until all 400 XMLs get submited.
If I have the limit set to more than 1 Run Per Thread, the 2nd Run will start after all 400 XMLs responses return. The 2nd run will repeat everything in the 1st run.
The Test Delay is the delay between each Run.

FIXED-RATE:
I have no idea other than knowing that it will not wait for the request to come back before submitting again. But where is this "wait" they are talking about? Is it the wait between the "Runs per Thread" or the wait between each of the 40 threads submiting within the 400 XMLS (between the 1st 40 XMLs and 41-80 and 81-120 and so on)?
And what is the "Rate" edit field for? If I set it to 10, does that mean only 10 requests are being submitted per second? Or only 10 Threads at a time (users) per second?
If I do not use the "Request level" checkbox, what happens? The website state it is on th erequest level, not the test case level - what is the difference?

It would really help out if the online user's guide gave good solid examples what happends when just one variable at at time is changed. It explains things but with little to no examples.

4 Replies

Sort By
  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    11 years ago
    PeterKlim wrote:
    SIMPLE:
    When starting the Load test run, each of the 40 users start at the same time. Each does 1 request. Only after all 40 request return a response will each of the 40 users submits another respononse - this gets repeated until all 400 XMLs get submited.
    If I have the limit set to more than 1 Run Per Thread, the 2nd Run will start after all 400 XMLs responses return. The 2nd run will repeat everything in the 1st run.
    The Test Delay is the delay between each Run.

    In Simple strategy, it runs the number of threads with the specified delay between each thread execution until the limit is reached or the execution is cancelled. So, all threads don't start at the same time.
    For example: if you have 40 threads, with Test Delay: 10000, Random: 0.5, Limit: 1 Run per Thread - this means that it will run 40 times (each thread once) with 5-10 seconds delay between each thread execution. If you want to run it 400 times then either you need to change the threads to 400 OR change the limit to 10 Runs per Thread.

    PeterKlim wrote:
    FIXED-RATE:
    I have no idea other than knowing that it will not wait for the request to come back before submitting again. But where is this "wait" they are talking about? Is it the wait between the "Runs per Thread" or the wait between each of the 40 threads submiting within the 400 XMLS (between the 1st 40 XMLs and 41-80 and 81-120 and so on)?
    And what is the "Rate" edit field for? If I set it to 10, does that mean only 10 requests are being submitted per second? Or only 10 Threads at a time (users) per second?
    If I do not use the "Request level" checkbox, what happens? The website state it is on the request level, not the test case level - what is the difference?

    In case of Fixed rate, there will be fixed number (specified as Rate) of threads running every second, the total number of running threads might be anything between Rate and Max Threads. There is no "wait" in this case. Rate is the number of threads you want to run every second.
    In general, each thread is applied on test case level, If you want that the fixed number of threads should apply to only Request test steps (in test cases) and not to the test case then you can enable/check "Request level" check box. This way the fixed number of threads will be maintained for the request test steps and not for the test cases.

    I hope this helps.

    Best Regards,
    Prakash
    SmartBear Sweden
  • PeterKlim's avatar
    PeterKlim
    Contributor
    11 years ago
    Hi Prakash,

    In this post I am going to only write about SIMPLE. Because my observation is MUCH different than what you are saying should happen, so I just want to focus on one strategey at a time (and I had a long day )

    Mine runs 400 XMLS, not 40. Yes it runs 40 threads, which the tool tip states the "Threads" field is "Virtual Users". My Data Source contains 400 XMLs - so why would it only do 40 requests? I have it set to "Set to share this DataSource between running threads during a Load Test". So all 40 use up the 400 XMLS. If I uncheck it, then each of the 40 will do 400, so that is 40 X400 = 16000 XMLs. This is shown in the "HTTP Test Rquest" "cnt" column in the load test. And these counts (400 and 16,000) are all when Runs Per Thread are set to 1.

    Test Delay - you wrote this is the delay between each thread, but I saw it as between Runs. I set the "Runs Per Thread" to 2. I set the Delay to 10 seconds (no Random) and the random to 0. I counted how much time it took between the 1st Run Per Thread and the 2nd - it was 10 seconds. After each Run I saw that the 16000 XMLs were completed (when set not to ""Set to share this DataSource...") 32000 XMLs completed after the 2nd Run. The tool tip on the "Test Delay" field states "Sets the delay between each test run in milliseconds". It states between RUNs, not THREADS. If it were Threads, that is 40 X 10 seconds and that makes out to 400 seconds (per RUN). My test finished in much less than 400 seconds.
  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    11 years ago
    Hi,

    Your understanding about Simple strategy is correct. The "Test Delay" is the delay between each run. And you can add randomness to the delay using "Random".

    Also, I have created a task (SOAP-1011) for it on our internal issue tracking system, so that we add documentation for this.

    Regards,
    Shadid
    SmartBear Sweden