Thanks for the valuable feedback.But now my question is
1)"If is test for 150 Vusers,test is coming as passed but again running it twice or many times times then error arrives at same 150 VU users.So how can I judge that 150 user are handled by the site or not because sometimes 60% errors are appearing sometime 0 at the same 150 Vusers.
One more question is that:
2)Some time at 200 vusers, no errors are arrived but sometime at 150 users errors arrived,then what will I consider?What is the point in giving error in 150 vuser but not in 200 vusers.
3)And to check the crash point, should I add some other parameters(time/duration of test,loop count etc.) or only by putting virtual users,we'll get to know the crash point.
4)And last,say if on 1000 vusers no error is coming,then on the basis of this test,we can take a decision that it can handle more than 1000 users?Is this test can be solely trustworthy that it can easily handle 1000 users at a time.?
Hi,
I hope that Ryan will correct me where I am wrong and add his knowledge, but here is my opinion:
-- Load tests are not deterministic but more statistical ones. This is because they are driving the complex system that consists from several components each of which is controlled separately and independently (e.g. data server is controlled by its own runtime engine that is absolutely not aware about web server engine and vise versa and thus those two engines do not coordinate their activities) and that communicate via commutated lines. The above means that the fact that the tests passed once for 100 VUs does not guarantee that it will definitely pass for another time. Instead, you should say something like: "With the probability of 95% 100 VUs will be able to browse the list of our products without any problem. For the remained 5%, not more than 10 VUs out of 100 (i.e. 10%) might not get the image of this or that product from the first attempt and not more that 2 VUs out of 100 might not get a list of products at all";
-- If the deviation of the number of failed VUs is small and the number of failed VUs itself is not big, then you may consider that the current number of VUs is the current limit for your system. I.e. for the system as a whole, which means that, for example, the bottleneck might be not in the web server, but in the communication channel between web server and the database;
-- If the deviation of the number of failed VUs is big for the same number of VUs (i.e. 2 VUs out 100 fail in one test run but in another test run 20 VUs fail and so on), you must examine the whole tested system and find out what component causes this or that failure. Here you might require the help from developers, network guys, etc. because the failures might be caused not only by the problems in your developed code, but, for example, the garbage collector service might start and cause the delays that lead to failures. So, without the knowledge of what causes some VUs to fail, it is not possible to answer whether or not the given number of VUs be served by your tested system and why, as per your example, test for 150 VUs may fail while the same test for 200 VUs will pass.
