Forum Discussion

leeasdf9's avatar
leeasdf9
Frequent Contributor
8 years ago

performance measurement with high accurrency requirement

Hi,

We have a performance requirement for our tested application that requires performance with high accurrency such as:

 

Test steps:

1. click button to lauch application

2. aqPerformance.Start()

3. wait until application is visible

4. aqPerformance.Stop()

 

Expected result:

appliation is lauched within 0.1 sec.

 

The question is whether testcomplete is suitable for such performance check? As far as we know its impossible to get the value under 0.1 sec with testcomplete.

Because object spy also needs time to detect object on step 3, it takes at least 0.5 sec.

Can somebody tell me whether TC is designed for such high accurrency performance checks.

 

Br,

Lee

 

15 Replies

Sort By
  • baxatob's avatar
    baxatob
    Community Hero
    8 years ago

    Hi,

     

    The order should be:

     

    1. aqPerformance.Start()

    2. click button to lauch application

    3. wait until application is visible

    4. aqPerformance.Check(100)

    • Colin_McCrae's avatar
      Colin_McCrae
      Community Hero
      8 years ago

      Should be fine for what you need.

       

      The documentation says it goes down to milliseconds.

       

      https://support.smartbear.com/testcomplete/docs/reference/program-objects/aqperformance/value.html

       

      Although, you have to factor in the time it takes to find you visible object. How much time that will add is hard to say. It depends what you're searching, how you search for it, and possibly how busy the applications object model is.

       

      If it starts any processes or services, it may be slightly quicker to check for those instead? Not sure. You'd need to have a play about with it and see for yourself.

      • Colin_McCrae's avatar
        Colin_McCrae
        Community Hero
        8 years ago

        I'll also add (as I do performance testing as well), that unless you are only running this application on specific, controlled, hardware, then such a measurement cannot be accepted as a global given anyway.

         

        Because:

         

        Run it on a high spec i7/SSD based PC ..... launch is super fast.

         

        vs

         

        Run it on an Atom based machine, clogged up with other rubbish installed, and running off a 5400rpm HDD, and the result will be much slower.

         

        Of course, if you control the hardware, then fine.

    • leeasdf9's avatar
      leeasdf9
      Frequent Contributor
      8 years ago

      Hi Baxatob,

      Thanks for your input, in my Case I want to measure the exact time how long the tested app. takes to be visible after button click, I still think my order is correct, in your order you also involve performance time of click event.  

      Kind Regards,

      Lee

      • Colin_McCrae's avatar
        Colin_McCrae
        Community Hero
        8 years ago

        exact time how long the tested app. takes to be visible after button click on the current hardware setup

         

        I fixed that for you. :smileywink:

         

        (Reasoning outlined further up. Trust me, I've dealt with people moaning about application performance only to discover they're running it on junk hardware with about a million other things running at the same time .....)

  • AlexKaras's avatar
    AlexKaras
    Champion Level 3
    8 years ago

    Hi,

     

    Been absolutely in line with all said by Colin and Robert and taking the fact that you did not mention type of application (desktop, web, mobile) and intended environment, I would add the following:

    -- For the desktop application in the Windows environment, you may consider AQTime (https://smartbear.com/product/aqtime-pro/overview/) that is a tool for profiling different aspects of the tested application. In your case, you may talk to developers and ask them at what moment the application is in the required state. (For example: return from the OnShow procedure for the AppMainForm garantees that the application is displayed and is ready to interact with the user.) Then you may use Performance profiler from AQTime and measure time from application start until return from the OnShow method. This should answer your question (obviously, as it was mentioned, for the given test run conditions). As a bonus, you will get a trace of what methods were executed before OnShow and how long did it take them to execute. So, if it takes too much time for your application to start, you will be able to answer what methods need be optimized first. Again, do not forget that all performance measurements are not absolute but rather statistical ones. You must perform measurements a couple of times (dispose of of any statistically incorrect results), process results and the answer should be like: "On the <Envoironment1> environment, the time from application start until end-user can communicate with the application is 0.1 sec with 95% probability".

     

    -- For web applications, you may consider PerformanceTiming API and measure time interval between navigationStart and loadEventEnd events on web page. But again, remember that a lot of web pages become responsive not after they were obtained from the server, but after the given page was rendered by the browser and all scripts with additional web requests were executed and their results were processed as well. LoadComplete (https://smartbear.com/product/loadcomplete/overview/) might be a tool that will help you to measure the peformance of your web application and how the performance degrades under load.

    • leeasdf9's avatar
      leeasdf9
      Frequent Contributor
      8 years ago

      Hi Alex,

      Many thanks for your inputs and sharing your deep knowledge, I really appreciate that.

      I tried also LoadComplete, but I think LC is a specialist for backend performance testing and measurement for http request/response, like you said, even requests were completed and responses were received from server, it still does not mean that browser/webapp. is responsive and applicable on gui. 

      As you said AQTime is maybe more suitable for this testcase, maybe I will try it out later.

      Thanks again and Kind Regards,

      Lee

  • leeasdf9's avatar
    leeasdf9
    Frequent Contributor
    8 years ago

    the precondition is that the test application is launched immediately on gui without delay. we want to proof with a measurementstool(e.g. Testcompete) that returns a high accurrency value such as 0.1 sec.