Forum Discussion

JimL's avatar
JimL
Contributor
14 years ago

Problem with test case under load test (repeating?)

Hi,

I have a test case with two test steps:

- A Groovy test step, which populates some properties
- A SOAP request, which uses the properties from the first test step via property expansion

Among the properties that the Groovy test step populates is a "Id" attribute in a SAML assertion in the SOAP message, and the Groovy test step is basically incrementing that "Id" property each time the Groovy test step is run.

If I manually run the test case (by clicking the green arrow on the test case window) several times, I can see the "Id" attribute incrementing by 1 each time, both in our server logs and in the test case Properties window. That's good.

So then I created a load test, using that test case. For now, I'm using the "Simple" strategy in the load test.

When I run the load test, and check the server logs, I see some/many messages with the same "Id" attribute, repeated. Also, at the end of the load test, I checked the server logs, and there are about 2000 requests, but the highest "Id" value I have is 79, whereas it should be some more on the order of 2000+.

I've been searching the forum, and noted this thread:

viewtopic.php?f=5&t=2218&p=8191&hilit=test+case+sequence#p8191

and where it says "The TestSteps of an individual TestCase are always executed Sequentially...".

But, based on what I'm seeing, it seems like that is not the case, i.e., it looks like the Groovy test step is not always being invoked before the SOAP request test step. In other words, it looks like the Groovy test step is executed, but then multiple SOAP request steps are being executed after that, before the Groovy test step is executed again (and incrementing the "Id" attribute)?

FYI, I'm currently using SOAPUI non-Pro V4.0.0 for this testing, downloaded and installed about a week ago.

Thanks,
Jim
  • JimL's avatar
    JimL
    Contributor
    14 years ago
    Hi,

    I think that I've figured out why I was seeing the same "Id" for several test case runs in a load test.

    It was because I had "Threads" in the load test set to 20. That would cause 20 requests to have the same "Id".

    When I set "Threads" to 1, I got unique "Id" values.

    This does, however, reduce the overall TPS of the load test.

    I'm guessing that in a load test, the test cases are running in individual thread, and that the properties for each thread are separate from each other. Since the properties my Groovy script is using are all test case properties, I get the repeated "Ids".

    I'm wondering if there's another 'level' where I could put the properties (e.g., the test suite level) that would allow all the threads in the load test to use the same properties, so that I could have more than 1 in the "Threads" setting, but still get unique Ids?

    I'm guessing that the answer to that is "no" because then ALL of the properties that my Groovy script uses would be shared among all the threads in the load test, and things would get messy (i.e., it would break things)??

    Jim
  • Finan's avatar
    Finan
    Frequent Contributor
    14 years ago
    (don't know if it is possible in soapUi free version)
    What I'd recommend is to create a testCase called "id_generator" in a different testSuite, in my case called: "Blocks_loadTest_functions".
    Everytime this testcase runs, a new id is generated and returned to a testCase property. (Id)

    You can add a runTestCase testStep in your loadTestCase, set the property id as return value and set the runmode of the testStep to "SINGLETON_AND_WAIT" (First selectable option).
    This will still be a (minimal) delay in your loadTest, since multiple threads cannot acces this testCase at the same time, but at least you can run multiple threads.
  • JimL's avatar
    JimL
    Contributor
    14 years ago
    Hi Finan,

    So, your "id_generator" acts as a kind of semaphore/mutex, but also generates a unique id each time it is run?

    Re.

    You can add a runTestCase testStep in your loadTestCase, set the property id as return value and set the runmode of the testStep to "SINGLETON_AND_WAIT" (First selectable option).


    When creating a new run test case test step, I can see where I can select return properties, but I don't see anywhere to set "SINGLETON_AND_WAIT" when creating a new test step?

    There's a setting for "Run Mode", with 3 choices (radio buttons):

    - Create isolated copy for each run (Thread safe)
    - Run primary Test case (fail if already running)
    - Run primary Test case (wait for running to finish, Thread safe)

    Is the "SINGLETON_AND_WAIT" one of the above 3 Run Modes?

    Thanks!

    Jim

    P.S. We're using non-Pro SOAPUI...
  • JimL's avatar
    JimL
    Contributor
    14 years ago
    Hi Finan,

    As an alternative to the approach you suggested, I'm still wondering if putting the properties at, say, the project level instead of at the test case level, would allow the properties to be "shared" among the test cases running in the threads that SOAPUI spawns?

    Do you happen to know?

    Thanks,
    Jim
  • JimL's avatar
    JimL
    Contributor
    14 years ago
    Finan,

    FYI, I was able to get a working solution, using another approach. What I'm doing is to include the context.ThreadIndex() and the system time in msecs in my "Id" attribute. That make the attribute unique among all the threads.

    Thanks for your help.

    Jim
  • Finan's avatar
    Finan
    Frequent Contributor
    14 years ago
    That should also do the trick;)

    And to answer your question:
    - Run primary Test case (wait for running to finish, Thread safe) == SINGLETON_AND_WAIT (this is the state in groovy:))