Forum Discussion

Colin_McCrae's avatar
Colin_McCrae
Community Hero
12 years ago

HUGE performance hit using TestExecute 10.10

Hi



I updated to TC/TE version 10.10 yesterday in order to retain compatibility with Chrome 34.



I'm now seeing a HUGE performance hit compared to version 10.0.



I run TestExecute on a dedicated VM as part of build process. TestExecute is the only thing that was updated on the VM. I have not yet updated Chrome to v34 on the VM, only on my local machine.



Running a set of tests I had run previously in the week, I can see the performance degradation quite clearly in my logs. Running exactly the same test as on Monday, the performance seems to be pretty much 100% worse.



For instance:



Example 1



Six steps which opens a text file, extracts the JSON stored in the file and sends it via HTTP to a webservice. So no onscreen activity of any description. This all happens within TestExecute.



On Monday, these six steps took 2 seconds.

Today, the same six steps took 5 seconds.



Example 2



The six steps mentioned above form part of a larger test. This test involves opening multiple browser instances and ensuring that the JSON sent to the webservice causes certain events to happen at the client end. It runs twice. Once against Chrome, once against IE.



On Monday - Chrome version took 8 minutes 20 seconds to run. Today it takes 16 minutes 49 seconds. IE version took 6 minutes 36 seconds to run. Today it takes 12 minutes and 2 seconds.





The only thing that was updated on that VM was TestExecute. All other software installed was untouched. The code for the test run (it's all script units) and data used during the test is all stored in version control and pulled down by the build process at run time.



So everything is identical except TestExecute.



The full test run on this VM was around 6 hours worth overnight. With a performance hit like this, it will push that time out to more like 12 hours. Which is way too long.



Any suggestions?



This needs urgent attention. I'm going to have to remove parts of my test run as it now overflows massively into the working day. We run overnight so we have results daily against the overnight builds ......



(I haven't tried the old favourite of un-install and re-install yet. But I'm about to. I'll post back if it makes any difference ....)

12 Replies

  • hlalumiere's avatar
    hlalumiere
    Regular Contributor
    12 years ago
    My advice from experience is if you find a TC release you don't have a problem with keep it, never update it. Just update the browser plugins as they come out.

  • Colin_McCrae's avatar
    Colin_McCrae
    Community Hero
    12 years ago
    Interestingly, I just tried a much smaller run on my local machine which I had a result for from earlier this week. This is using the full version of TestComplete.



    Virtually no change. Run time on Monday was 3:51. Today it was 3:54.



    Next step is clean out and re-install TestExecute and see if it makes any difference ....
  • Colin_McCrae's avatar
    Colin_McCrae
    Community Hero
    12 years ago
    Oh dear.



    I un-installed TestExecute and did a clean install.



    Have only had time to wait for one part of the test suite to complete before I leave for the day but, if anything, it's now even worse!



    The chrome test I mentioned in the previous post:



    v10.0 - 8:20

    v10.10 before reinstall - 16:49

    v10.10 after reinstall - 20:59



    Nothing else in the VM has changed. I have sole use of it. Nothing else is running at the same time. All applications, test code/steps/data are all identical in all three runs.



    This effectively makes TestExecute unusable for me. I'm going to have to rewind everything back to v10 and force Chrome to freeze on a version earlier than 34.



    Not good.
  • simon_glet's avatar
    simon_glet
    Regular Contributor
    12 years ago
    Hi Colin,



    Your VM might be same but what about the host supporting all the VM ? As much as TE 10.10 could be buggy, I would first check what are the available ressources.



    If there is some intense processing going on some free running VMs, it could seriously impact all the hosted VMs.

    If the RAID5 disk array has a failed disk and is rebuilding on the spare, that too will seriously impact the VMs performances.

    If a test is flodding the network, etc etc ;-)



    I mean as this is for some of us the last day before a long weekend, why not launch everthing that we can't run during normal weekends ?



    Sincerely



  • Colin_McCrae's avatar
    Colin_McCrae
    Community Hero
    12 years ago
    No.



    These are dedicated VM's hosted within our datacentre for build use only.



    The one I use was created within this environment exclusively to run TestExecute as part of the nightly build process. I've checked activity within the VM and there is nothing running in there except what I'm running.



    Environment change was the first thing I wondered about so I also checked with the build engineer who maintains all this and there has been no change or extra activity. It's all as it was pre-update.



    And I'm seeing this performance hit on all aspects of the run. Example 1 above illustrates it clearly.



    The problem is definitely with TestExecute. I'll double confirm that tomorrow as I'm going to have to roll everything back to v10 to make it usable again.



    I've also raised a support ticket for this. Ticket number is 00034194.
  • simon_glet's avatar
    simon_glet
    Regular Contributor
    12 years ago
    Sorry I couldn't help.



    In any case, please keep us updated when you figure it out. 



    We are contemplating a TC-TE update from 9.2 but we are waiting for a release praised by all !



    Sincerely

  • Colin_McCrae's avatar
    Colin_McCrae
    Community Hero
    12 years ago
    No worries. Thanks for the reply. :)



    I'll see what happens when I roll back tomorrow. And keep this thread updated with the outcome of the roll back as well as any feedback I get from the support ticket.
  • Colin_McCrae's avatar
    Colin_McCrae
    Community Hero
    12 years ago
    Have reverted the build VM back to v10.0 and immediately everything is back to running at the previous speeds.



    So the problem is with the TE 10.10 update.



    Looks like I'll be staying on 10.0 for now ....
  • leandropoblet's avatar
    leandropoblet
    Frequent Contributor
    12 years ago
    Hi all,



    so here are my results supporting the idea of TestExecute being ridiculously slow.



    Attached you'll find the summary of different runs of the same project (189 steps) in different scenarios:



    TC_Run_1_All_Plugins:

         > this is TestComplete running with all plugins (default installation).

        > Took 11m 40.220s



    TC_Run_2_Used_Plugins_Only:

         > this is TestComplete running just with the needed plugins.

        > Took 11m 15.737s

        > As you can see in this case turning off the unnecessary plugins gave me a thrilling 30 seconds faster every 11 minutes. Not to bad.... not too good either



    TE_Run_1_Win7_VM_1:

        > this is TestExecute running on a VM (win7). It's got 2gb of RAM and 1 core (enough for running our application even 2 instances)

        > Took 2h 20m 21.473s

        > As you can see the plugins factor doesn't affect at all TE performance.... it's just unacceptable



    TE_Run_2_Win8_VM:

        > again TestExecute running on a VM (win8). Can't recall RAM and cores, but they are more than enough for our porpoises. This VM is running in an i7 build.... so there's no lack of gear in here at all.

        > Took 3h 17m 18.843s

        > Totally unacceptable and actually worst record so far



    TE_Run_3_Win7_VM_2:

        > TestExecute running on a VM (win7).

        > Took 2h 17m 33.604s





    Just to clarify the project we're running for testing just the login system of our application. I mean.... is probably the easiest and fastest test developed so far so imagine what happened if I run the whole roject. I just can wait that long.



    Please do contact me if you have any questions, but basically I need a better solution than turning plugins off.



    Waiting for comments,

    Leandro

  • leandropoblet's avatar
    leandropoblet
    Frequent Contributor
    12 years ago
    Hi all,



    second time writting this post (don't try to attach files and go back.... forum is not intelligent enough to save the text)



    All right so I run this really easy and fast tests on different scenarios to measure how bad TestExecute performs and here are the results. It's basically testing the login system of our application, not big deal in terms of difficulty.



    Attached you'll find the actual screenshots in case you want to double check.



    TC_Run_1_All_Plugins:


    • TestComplete v10.20 running with all plugins (default installation)


    • It took: 11m 40.220s to complete the 189 steps




    TC_Run_2_Used_Plugins_Only:


    • TestComplete v10.20 running with just the used plugins to measeure how much plugins affect performance


    • It took: 11m 15.737s to complete the 189 steps


    • This means we got a performance improvement of 30 seconds let's say every 10 minutes. Not too bad. Not good either




    Now TestExecute:



    TE_Run_1_Win7_VM_1:


    • TestExecute v10.20 running on a VM (Win7, 1 core, 2gb RAM enough for our purposes)


    • It took: 2h 20m 21.473s to complete the 189 steps


    • Awful result. More than 2 hours more to complete the same tests.....




    TE_Run_2_Win8_VM:


    • TestExecute v10.20 running on a VM (Win8, 1 core, 2gb RAM enough for our purposes)


    • It took (grab a seat): 3h 17m 18.843s to complete the 189 steps


    • This is by any means an acceptable result.


    • One more thing: no unnecessary plugins where used of this test




    TE_Run_3_Win7_VM_2:


    • TestExecute v10.20 running on a VM (Win7, 2 core, 2gb RAM on an i7 build)


    • It took: 2h 17m 33.604s to complete the 189 steps




    I'm really looking forward to receive some explanations on what's going on here. If this is the kind of performance we'll get from now well.... I don't know



    Please share more results so we can compare.



    Cheers,

    Leandro