TestExecute vs TestComplete Execution Speed

[kirk_bottomley]

In our environment, I write the automations on one system, then we use our TE licenses to execute them on other machines (or I run it in TC on the machine they were written on). We are just now starting to do these distributed tests, so this is the first time we've actually used TestExecute.



I completed a rather lengthy test, and tried running it on another machine with identical hardware. The test runs in 90 minutes on my machine, it takes nine hours to run in TE on the other.

I loaded up TE on my machine and ran with TestExecute instead of TC, and it only took the 90 minutes. 



I have reviewed the tips at http://support.smartbear.com/articles/testcomplete/test-playback-performance-tips/ , but nothing there helps.



Any ideas why  TE runs faster on the machine that has TC installed, but many times slower on identical hardware that doesn't?

[chrisb]

You mentioned that the machines are 'identical' and that you believe the only difference is that the faster running suite is on the machine with TC installed as well as TE. Did you try installing TC on the 'problem' machine running TE to see if this theory is correct? 



From my experience of using TE I don't think having TC installed on the same machine makes any difference. Assuming  having TC installed on the 'problem' machine doesn't speed up your tests in TE I would then do some more investigation on the problem machine.



Also, what kind of app are you testing? web, desktop, mobile? Is there something that can be a bottleneck on the problem machine such as network bandwith, accessing databases. reading / writing data to somewhere?



I'd also do some debugging of the tests using the Test Complete IDE on the problem machine if you are able to install it there.Ie. step through some slow parts of test and see if you can narrow down 'problem' areas in the test.

[kirk_bottomley]

Identical, as far as make, model and OS.



I can't install TC on any other system. The TC license is node locked. The TE licenses are floating.



I tried running it on a TE license in a VM, and it took about nine hours. I figured it was a VM thing. When we tried running it on native hardware, it still took nine hours. The 90 minutes was conservative, too. I just ran it again using TE on the same machine as TC, it took 36 minutes.



The tests are run on a desktop application, on Windows 7. The test files are networked for all machines, including the authoring. The project location is also networked, for all machines. So in theory, any network lag should be the same even if it's run on the TC machine.



It could very well come down to a configuration difference, but the similarity between the VM and the hardware run time made me think this was an issue someone had seen before.

[chrisb]

Doesn't fix your issue but I am writing tests in TC on a desktop and running them on a vritual machine using Test Execute and I dont see a difference in test execution time. In theory my desktop and the VM are both Windows 7 and the same configuration. If anything the VM may even have less resources than my desktop. 



Does your version of Test Execute align with your version of Test Complete?

[kirk_bottomley]

Yes, same version. The TE instance on the other machine is also the same version.

[dganov]

Try to compare test execution logs to see if every test action runs slower or there are some specific actions that slow down the test, or maybe you will find some other differences.

[chrisb]

Kirk, after replying earlier I remembered an issue I saw with slow running tests on a virtual machine in a past project. We were using a different tool (Sikuli) and I found that tests ran really slow on my VM. 



I monitored the VM resources in realtime using the virtual machine manager and found that the VM was maxed out. Try running the tests again on your VM and monitor the machine resources in realtime. You might be suprised. The VM might just need more resources allocated to it. In our case we were using VMWare ESX server and had multiple VM's running concurrently and thus dividing the resources between the machines running on the one server. Hope this helps.

[maximojo]

Definitely look at the logs and see if there's a choke point where most of the time is taken up.



I have tests where I export the test results to a webserver for viewing over the intranet. when I run my tests on the webserver machine MACHINE_A (it does triple-duty as test runner, webserver and db host) and export the results they export reasonably quickly as, obviously, the directory they are exporting to lies on that actual machine.



But when I run those tests on other machines and they have to export to the webserver directory on MACHINE_A they take a whole lot longer as they have to export across the network.



Just one example of why a test run could take longer on another machine.

[AlexKaras]

Hi Kirk,



I am not sure if this is relevant, but maybe a piece of advice from Hugo will help somehow...

http://smartbear.com/forums/f81/t83791/testcomplete-limitations-on-ms-virtual-server-o/

[kirk_bottomley]

I've tried all kinds of things. I'm still getting the same issue. 

I've installed TE on VM's, servers, and desktop hardware. I point to the same project and run the same test. If it's run using TC or TE on the machine I use to write the tests, everything is fine. If I run the tests using TE on any other machine, virtual or otherwise, it runs each step excruciatingly slow. 

The only information I've been able to get from the logs is that each step is excruciatingly slow. 



I have no idea what to try next. I've checked admin rights, network rights, installation of TE with full admin rights.. I'm at a loss. We can't move forward with distributed testing until this is resolved.