Test quit parameters ? How we know a server's capacity from results ?

Hi Johny,



I am not sure that the statement like "my customer wants to learn how many  user can connect and use their system concurrently. So, this is the requirement." can be considered as the requirement. The reason for this is that requirement must contain the *measurable expected result* which is absent in your case. So, instead of requirement, I would call the above statement a "question to investigate".

Considering that you have the given fixed operational environment, I think that the only thing that you can do is to measure response time for different loads (number of virtual users) and present the obtained data to the customer. Considering that you don't have business requirements, you cannot give an answer to the "how many  user can connect and use their system concurrently" question (unless all requests start to fail after some number of virtual users is reached). I think that you cannot answer the question because you don't have evaluation criteria (except that the user cannot login). Returning back to one of my examples: Is the system usable if the response time is 20 minutes? Definitely no if we are talking about online auction. But quite possible that it can be used if this is the time needed to generate an annual summary tax report for one accountant only.

So it is customer who, using the figures provided by you, can make a decision of how many users can be served by the given system.

The above approach is quite a normal one as one of the tasks of the tester (actually - this is the task for QA during the requirements analysis stage) is to analyze what can be tested and what cannot and communicate with the relevant people (customer, analyst, etc.) to discuss why some statement cannot be tested, what needs to be chainged and what can be done for this statement in its current wording.



Also I think that you should communicate with the customer and project management and work out the activities for this case. This is because I think that this is not correct that developers do not care about load capabilities of the product and only system administrator is considered to be responsible person. Well, it is possible that in your case this is responsibility of the system administrator, but usually what system administrator is responsible for is to guarantee that the server software (i.e. web server as a system service) is running and restarted in case of crash, that the CPU and memory of the server are not overloaded, that the network can service requests and things like that. It is not his problems that the software cannot handle too many or concurrent requests, it is not his problems that the code is not optimized and so on. These are the problems of development and system administrator can only help you and development to figure out where the problem occurs (application code, overloaded resources, congested network, etc.) and whether and what problem was reported by the system to one of the system logs.

You should not expect the TestComplete will provide you with the information about the reason of the problem. It just cannot do this, because HTTP protocol is a stateless one and it only assumes that the server receives request from the client and sends response and the status code back. It is up to the server whether to include or not the diagnostic in the response in case of the problem. If nothing is included in the response, than TestComplete have no means to figure out why the request failed and this is your task (together with development and network specialists) to figure out what the problem was and why it occurred.

 

Thank you,

I got the answer except few points;

The first one is about response times, 



when you wrote

 "measure response time for different loads (number of virtual users)"

 what di you mean by response time ?

 When I look at the log page, I only see Response Time in the request panel. 

So Do I have to check for an exact request, and compare for every test ? For mhow many vuser ?

Or , you are talking about total resonse time ? if so, how will I get this value ?



Can you tell me in a more spesific way to measure this value and how to compare ?



The second point is that, "how many  user can connect and use their system concurrently" question (unless all requests start to fail after some number of virtual users is reached)"



So at this point, two cirital words that is not clear for me; number of request failed, and the "fail" word.



In my test, not all but when I start to increase, few requests start to fail, so which percentage is dangerous



and, what is failing ? there is no connection "error", just warning for 500 internal error.

And for 500 status code error, we can not give our customer detailed answer when they want to learn real reason for the problem..



and you say that "delay time" is depend on web-site owner decision, and customer  reaction



So, what it has to be happen to call it a fall ? For example, if 50 vusers come with 500 status code warning, of total 100 vuser,   is this a fall ? is

Hi Johny,



You did not provide any information about the tested application, so my answers and considerations are quite generic ones.



>  what di you mean by response time ?

I think that this should be specified in the requirements for the test as the test plan / test design should define what should be measured and how. For example, it may be required that *any* request must be responded within not more than one second. In this case you should measure the time for every request/response. Alternatively, the requirement may be that it must take not more than one second for the end-user to see the page in the browser. In this case you should sum the time for all requests/responses that are executed for the given page(es) (also consider the time that it takes for the scripts to execute on the client's computer) and check that the total does not exceed required one second. In this case it does not matter how long does it take for one separate request/response to execute but only the total should be counted.



> [...] number of request failed, and the "fail" word.

> In my test, not all but when I start to increase, few requests start to fail, so which percentage is dangerous

I understand failure as TestComplete treats it - this is the difference between recorded and actual result code for the response.

What failures consider to be critical and what percentage is dangerous - again, depends on the requirements and what exact requests failed. For example, it may be acceptable that the server fails to return some logo images under high load. But most probably it is not acceptable if the user cannot login. In the latter case, there must be requirement that specifies what percentage of login failures is acceptable. For example, it may be required that up to 50 users must be able to login *simultaneously*. And it may be acceptable that no more that 5 login failures may occur if the number of *simultaneous* logins is between 51 and 200.



> And for 500 status code error, we can not give our customer detailed answer when they want to learn real reason for the problem..

And this is, actually, the task for you and/ or project team. You must figure out the reason of this problem - whether this is TestComplete that fails at some point, or this is a failure of the tested application code executed on the server, or this is the problem of the server itself (e.g. low memory), etc. And only after you find the reason of the problem you will be able to provide the customer with some definite information like: 'there is a problem in the application code and it will be fixed in two weeks', or 'do not allow more than 500 concurrent users otherwise they will not be able to login', etc. In the worst case, if you fail to find the reason of the problem, you may, for example, recommend customer not to execute application on the server with less than 2GB of RAM and limit the number of concurrent users to 300, though I am not sure that the customer will be completely happy. But anyway, some information is better than no information at all. :)



 

Hi Alex,



Actually the requirements in my test are not specified very detailed.

You gave me two case, but if the customer just want to test the system for a big customer increase in near future.

Like every B2C web site, they want en estimated users to connect at sime time and get service, of course with acceptable response times for the customers.In which case you can put in this situation ? I thought I can put in both cases.



But of course the site do not have to respond in same seconds for every request, there will be different response times for different actions.



The information I need here is the exact way to calculate response times. In both cases you defined.



Lets think about first case, which you defined as "In this case you should measure the time for every request/response."

How will I make it if there are hundred request in a task ? by using scripts.

and if you look my early posts, I also wonder how to calculte response times for an action, for an connectione etc.



In an article which named "Load testing of Web Sites" I find this definiton for response time,

"Another key QoS factor for Web and e-commerce sites is response time. With Web-based environments, you must measure end-to-end response time to determine how customers perceive such things as page download and keyword search times. Also, when defining end-to-end response time, you must distinguish between the download time for the base HTML page and that for other page components, such as images and ad banners."

In log results, I can see response time for only requests, and for connections, I can see

Request Execution Summary (s) ="The overall time that it takes to execute all of the requests of the given connection



So, for a spesific task, let say that the login time, or invoice address getting time,

How will I calculate time ? Just sum up all related request times, or or related connection times ?



The other point is that,

"And this is, actually, the task for you and/ or project team. You must figure out the reason of this problem"

Yes of course it is my task, I must figure out reason, and I am asking you how will I figure out..

Hi Johny,

So, for a spesific task, let say that the login time, or invoice address getting time, How will I calculate time ? Just sum up all related request times, or or related connection times ?

Again, it depends on your test requirements, actually. However, I can specify a few sample requirements for you if you want. You can either calculate the average value by summing up Request Execution Summary values for all the connections you need and dividing the resulting value by the total number of connections - in this case, the requirement will be: "the average time must not be greater than N seconds". Or, you can check the sum of Request Time and Response Time for some specific requests - in this case the requirement will look like this: "The X, Y and Z actions which are performed via requests #1, #2 and #3 must be performed within N seconds. If at least one action takes longer, the test is considered as failed".

"And this is, actually, the task for you and/ or project team. You must figure out the reason of this problem" Yes of course it is my task, I must figure out reason, and I am asking you how will I figure out..

As I understand, you've found that out in the following thread:


http://smartbear.com/forums/forum/post/?mode=singleThread&thread=ec7171a4-798f-4227-af5d-5a68c4f8d06f

Hi Allen,

ok lets say it is fine up to that point. But Request Execution Summary by itself does not give some much thing.



When I compare response times vs time-taken value from IIS logs,  I see Response Times are much more smaller than those IISs  time taken.



Can you compare these two ?

Hi Johny,





Please provide us with screenshots showing the IIS results and TestComplete results so we can see the difference.

Ok Allen, instead o screenshots, I will send you small log files via support, but I still dont know what these two value exactly corresponds to.

Hi Johny,





We'll be waiting for the files.

Hi

any update can be shared? I am facing problem on analyzing load test result as well.

Also, I think TestComplete should provide graph as below link as basis because most of the load test article also use these graphs as example. It made us easier to understand



Response time vs. user load

http://i.msdn.microsoft.com/dynimg/IC134036.gif



Throughput vs. user load

http://i.msdn.microsoft.com/dynimg/IC98242.gif



Utilization vs. user load

http://i.msdn.microsoft.com/dynimg/IC93235.gif



I hope TC can provide more article to show how to use TC to analyze the performance instead of just let us know how to use the tool. Like now, we know how to use it, but don't know how to analyze it, what does the result mean? In what situation the graph will be shown so and so.