Forum Discussion

johnc_1's avatar
johnc_1
New Contributor
8 years ago

Maximum number of simultaneous scenarios to run

We have a client who is requesting that we run 50 different scenarios, simultaneously over the course of 8 hours, to simulate their real world activity.  These scenarios mostly would be 10-15 step sequences, all starting from a log in screen with significant security validations.

Before we go too far down this road, wondering if anyone can comment on what they feel is a typical number of scenarios to run is in order to appropriately test the system's performance.  Or, from the perspective of monitoring and evaluation of results, what would be the maximum to run at the same time.

Thanks in advance.

  • baxatob's avatar
    baxatob
    Community Hero
    8 years ago

     

    You are limited only by license that you have. Each virtual user can use its own scenario.

  • AlexKaras's avatar
    AlexKaras
    Champion Level 3
    8 years ago

    Hi,

     

    The following is my current understanding. Note, not all bullets were verified in a real-world life, might appear to be incorrect (at some points) and I will appreciate comments / corrections from others.

    -- Initial assumption is that user actions (read - scenarios) do not depend on on the previous actions of this user. For example, if the user of the tested automotive trading web application looked at Ferrari, it is out of our scope to deal somehow with the fact that he/she at the application's sidebar will be suggested also to take a look at brand new Lamborgini as well. Things like that should be verified via functional tests and in our scenario we will just know that one or several requests for the sidebar are executed along with the main request(s);

    -- Recorded scenarios must not be long and complex. The reason for this is that as long as the given web page evolves, the number of requests that it generates as well as the target of these requests may be changed by developers. So it should be easy for you to re-record the changed traffic;

    -- You may craft more complex scenarios by creating a new blank scenario and populating it with the already existing scenarios using the Call Scenario operation;

    -- With all required scenarios ready, consider their distribution in a real world using logs from your web server. E.g. 60% of visitors just browse for one car, 35% browse for two cars; 2% additionally look for the list of distributors, 0.2% do a purchase, etc. Based on this distribution, consider the number of virtual user groups, connection speed and browser for each group, the number of users in each group (http://blog.smartbear.com/performance/how-to-determine-your-virtual-user-quota/, https://smartbear.com/learn/performance-testing/load-testing-vu-calculator/ and https://www.google.com/?q=load+testing+required+number+of+virtual+users), what scenarios will be executed by this or that group and whether or not the test is worth to be distributed on several machines;

    -- Create Tests using the results of the above considerations.

    • johnc_1's avatar
      johnc_1
      New Contributor
      8 years ago

      Perhaps it might help if I provided some detail of our application.  It is a web based software package for the administrative side of K-12 schools...ranging from student and family demographics information, student data (attendance,  schedules, health, discipline, standardized test results, etc), teacher gradebooks (and attendance taking by teachers), family and student portal to the student information.  So, during the course of a day, the list of activities occurring that could reasonable need scenarios built include:

      1. Enrolling/registering students and  updating data - 200-300 users
      2. Maintaining and creating student class schedules - 200-300 users
      3. Discipline offense and response entry - 200-300 users
      4. Viewing student data in the administrative portal - 300-500 users
      5. Accessing/updating historical data such as grades, enrollments (db reads) - 200-300
      6. Teacher entry of absences in their class - 2000 (not all concurrently in groups, such as all H.S. teachers)
      7. Teacher creation of grading assignments and updating of assignments - 2000 users throughout day
      8. Reports of data - 100-300 users
      9. Viewing of student data with some update options by parents - 40,000 potentially over course of day
      10. Viewing of student data with some update options by parents - 75,000 potentially over course of day

      With that type of real world activity,

      1. While we can generate scenarios and parameterize entries, is it realistic to run such a test continually over an 8 hour day cycle?

      2. With at least 10 parameterized scenarios running continuously, will monitoring of the activity as well as interpretation of the results become too massive...other than to determine whether such activity reaches the max stress point.

       

      Does that help?...

      • AlexKaras's avatar
        AlexKaras
        Champion Level 3
        8 years ago

        Hi,

         

        > 1. While we can generate scenarios and parameterize entries, is it realistic to run such a test continually over an 8 hour day cycle?

        I don't expect any problem with this so far. https://support.smartbear.com/viewarticle/78727/ and https://support.smartbear.com/viewarticle/78821/.

         

        > 2. With at least 10 parameterized scenarios running continuously, will monitoring of the activity as well as interpretation of the results become too massive...other than to determine whether such activity reaches the max stress point.

        Client-side monitoring is described here: https://support.smartbear.com/viewarticle/78788/ and the server-side is described here: https://support.smartbear.com/viewarticle/78462/. Results analysis is described here: https://support.smartbear.com/viewarticle/78920/.

        The complexity of analysis depends on your application. Basically, if you run all your load tests within planned time interval and none of the tests fail, server replies were within acceptable time interval and servers monitoring do not show any unwanted trends (like memory consumption increase or disk access time problems) then you may feel (more or less;) ) confident.

        Though if the problems are noted, then you will have to figure-out their root and this might require your cooperation with developers, database admins, etc.