Forum Discussion
AlexKaras
15 years agoCommunity Hero
Hi Johny,
I am not sure that the statement like "my customer wants to learn how many user can connect and use their system concurrently. So, this is the requirement." can be considered as the requirement. The reason for this is that requirement must contain the *measurable expected result* which is absent in your case. So, instead of requirement, I would call the above statement a "question to investigate".
Considering that you have the given fixed operational environment, I think that the only thing that you can do is to measure response time for different loads (number of virtual users) and present the obtained data to the customer. Considering that you don't have business requirements, you cannot give an answer to the "how many user can connect and use their system concurrently" question (unless all requests start to fail after some number of virtual users is reached). I think that you cannot answer the question because you don't have evaluation criteria (except that the user cannot login). Returning back to one of my examples: Is the system usable if the response time is 20 minutes? Definitely no if we are talking about online auction. But quite possible that it can be used if this is the time needed to generate an annual summary tax report for one accountant only.
So it is customer who, using the figures provided by you, can make a decision of how many users can be served by the given system.
The above approach is quite a normal one as one of the tasks of the tester (actually - this is the task for QA during the requirements analysis stage) is to analyze what can be tested and what cannot and communicate with the relevant people (customer, analyst, etc.) to discuss why some statement cannot be tested, what needs to be chainged and what can be done for this statement in its current wording.
Also I think that you should communicate with the customer and project management and work out the activities for this case. This is because I think that this is not correct that developers do not care about load capabilities of the product and only system administrator is considered to be responsible person. Well, it is possible that in your case this is responsibility of the system administrator, but usually what system administrator is responsible for is to guarantee that the server software (i.e. web server as a system service) is running and restarted in case of crash, that the CPU and memory of the server are not overloaded, that the network can service requests and things like that. It is not his problems that the software cannot handle too many or concurrent requests, it is not his problems that the code is not optimized and so on. These are the problems of development and system administrator can only help you and development to figure out where the problem occurs (application code, overloaded resources, congested network, etc.) and whether and what problem was reported by the system to one of the system logs.
You should not expect the TestComplete will provide you with the information about the reason of the problem. It just cannot do this, because HTTP protocol is a stateless one and it only assumes that the server receives request from the client and sends response and the status code back. It is up to the server whether to include or not the diagnostic in the response in case of the problem. If nothing is included in the response, than TestComplete have no means to figure out why the request failed and this is your task (together with development and network specialists) to figure out what the problem was and why it occurred.
I am not sure that the statement like "my customer wants to learn how many user can connect and use their system concurrently. So, this is the requirement." can be considered as the requirement. The reason for this is that requirement must contain the *measurable expected result* which is absent in your case. So, instead of requirement, I would call the above statement a "question to investigate".
Considering that you have the given fixed operational environment, I think that the only thing that you can do is to measure response time for different loads (number of virtual users) and present the obtained data to the customer. Considering that you don't have business requirements, you cannot give an answer to the "how many user can connect and use their system concurrently" question (unless all requests start to fail after some number of virtual users is reached). I think that you cannot answer the question because you don't have evaluation criteria (except that the user cannot login). Returning back to one of my examples: Is the system usable if the response time is 20 minutes? Definitely no if we are talking about online auction. But quite possible that it can be used if this is the time needed to generate an annual summary tax report for one accountant only.
So it is customer who, using the figures provided by you, can make a decision of how many users can be served by the given system.
The above approach is quite a normal one as one of the tasks of the tester (actually - this is the task for QA during the requirements analysis stage) is to analyze what can be tested and what cannot and communicate with the relevant people (customer, analyst, etc.) to discuss why some statement cannot be tested, what needs to be chainged and what can be done for this statement in its current wording.
Also I think that you should communicate with the customer and project management and work out the activities for this case. This is because I think that this is not correct that developers do not care about load capabilities of the product and only system administrator is considered to be responsible person. Well, it is possible that in your case this is responsibility of the system administrator, but usually what system administrator is responsible for is to guarantee that the server software (i.e. web server as a system service) is running and restarted in case of crash, that the CPU and memory of the server are not overloaded, that the network can service requests and things like that. It is not his problems that the software cannot handle too many or concurrent requests, it is not his problems that the code is not optimized and so on. These are the problems of development and system administrator can only help you and development to figure out where the problem occurs (application code, overloaded resources, congested network, etc.) and whether and what problem was reported by the system to one of the system logs.
You should not expect the TestComplete will provide you with the information about the reason of the problem. It just cannot do this, because HTTP protocol is a stateless one and it only assumes that the server receives request from the client and sends response and the status code back. It is up to the server whether to include or not the diagnostic in the response in case of the problem. If nothing is included in the response, than TestComplete have no means to figure out why the request failed and this is your task (together with development and network specialists) to figure out what the problem was and why it occurred.