In our environment, I write the automations on one system, then we use our TE licenses to execute them on other machines (or I run it in TC on the machine they were written on). We are just now starting to do these distributed tests, so this is the first time we've actually used TestExecute.I completed a rather lengthy test, and tried running it on another machine with identical hardware. The test runs in 90 minutes on my machine, it takes nine hours to run in TE on the other.I loaded up TE on my machine and ran with TestExecute instead of TC, and it only took the 90 minutes. I have reviewed the tips at http://support.smartbear.com/articles/testcomplete/test-playback-performance-tips/ , but nothing there helps.Any ideas why TE runs faster on the machine that has TC installed, but many times slower on identical hardware that doesn't?

Kirk,This is a long shot. Our distributed tests have been running slowly on the VM's. Performance metrics did not show much activity of interest and the symptoms were pretty much those of an I/O bottleneck. Monitoring TestComplete10 with Sysinternals Process Monitor showed a large number of calls back to the developer machine. Doing a DOS findstr on the workstation name in project files showed that the VM's were querying the developer station for performance metrics. Removing the metrics got rid of the problem.Of course, the VM's are still slow compared to workstation, but performance is much better in general.

TestExecute vs TestComplete Execution Speed

mtsmith
Contributor
11 years ago
Kirk,

This is a long shot. Our distributed tests have been running slowly on the VM's. Performance metrics did not show much activity of interest and the symptoms were pretty much those of an I/O bottleneck. Monitoring TestComplete10 with Sysinternals Process Monitor showed a large number of calls back to the developer machine. Doing a DOS findstr on the workstation name in project files showed that the VM's were querying the developer station for performance metrics. Removing the metrics got rid of the problem.

Of course, the VM's are still slow compared to workstation, but performance is much better in general.
kirk_bottomley
Contributor
11 years ago
Good call, Mark! That did the trick.

For that particular project, I had turned on four performance counters that tracked the authoring machine's stats. Whenever another system opened the project with TestExecute, they were also trying to track the authoring machine's stats during playback.

I deleted the Performance Counters, and everything is smooth as silk now.

Thank you very much!
kirk_bottomley
Contributor
11 years ago
I've tried all kinds of things. I'm still getting the same issue.

I've installed TE on VM's, servers, and desktop hardware. I point to the same project and run the same test. If it's run using TC or TE on the machine I use to write the tests, everything is fine. If I run the tests using TE on any other machine, virtual or otherwise, it runs each step excruciatingly slow.

The only information I've been able to get from the logs is that each step is excruciatingly slow.

I have no idea what to try next. I've checked admin rights, network rights, installation of TE with full admin rights.. I'm at a loss. We can't move forward with distributed testing until this is resolved.
chrisb
Regular Contributor
11 years ago
You mentioned that the machines are 'identical' and that you believe the only difference is that the faster running suite is on the machine with TC installed as well as TE. Did you try installing TC on the 'problem' machine running TE to see if this theory is correct?

From my experience of using TE I don't think having TC installed on the same machine makes any difference. Assuming having TC installed on the 'problem' machine doesn't speed up your tests in TE I would then do some more investigation on the problem machine.

Also, what kind of app are you testing? web, desktop, mobile? Is there something that can be a bottleneck on the problem machine such as network bandwith, accessing databases. reading / writing data to somewhere?

I'd also do some debugging of the tests using the Test Complete IDE on the problem machine if you are able to install it there.Ie. step through some slow parts of test and see if you can narrow down 'problem' areas in the test.
kirk_bottomley
Contributor
11 years ago
Identical, as far as make, model and OS.

I can't install TC on any other system. The TC license is node locked. The TE licenses are floating.

I tried running it on a TE license in a VM, and it took about nine hours. I figured it was a VM thing. When we tried running it on native hardware, it still took nine hours. The 90 minutes was conservative, too. I just ran it again using TE on the same machine as TC, it took 36 minutes.

The tests are run on a desktop application, on Windows 7. The test files are networked for all machines, including the authoring. The project location is also networked, for all machines. So in theory, any network lag should be the same even if it's run on the TC machine.

It could very well come down to a configuration difference, but the similarity between the VM and the hardware run time made me think this was an issue someone had seen before.
chrisb
Regular Contributor
11 years ago
Doesn't fix your issue but I am writing tests in TC on a desktop and running them on a vritual machine using Test Execute and I dont see a difference in test execution time. In theory my desktop and the VM are both Windows 7 and the same configuration. If anything the VM may even have less resources than my desktop.

Does your version of Test Execute align with your version of Test Complete?
kirk_bottomley
Contributor
11 years ago
Yes, same version. The TE instance on the other machine is also the same version.
dganov
Staff
11 years ago
Try to compare test execution logs to see if every test action runs slower or there are some specific actions that slow down the test, or maybe you will find some other differences.
chrisb
Regular Contributor
11 years ago
Kirk, after replying earlier I remembered an issue I saw with slow running tests on a virtual machine in a past project. We were using a different tool (Sikuli) and I found that tests ran really slow on my VM.

I monitored the VM resources in realtime using the virtual machine manager and found that the VM was maxed out. Try running the tests again on your VM and monitor the machine resources in realtime. You might be suprised. The VM might just need more resources allocated to it. In our case we were using VMWare ESX server and had multiple VM's running concurrently and thus dividing the resources between the machines running on the one server. Hope this helps.
maximojo
Frequent Contributor
11 years ago
Definitely look at the logs and see if there's a choke point where most of the time is taken up.

I have tests where I export the test results to a webserver for viewing over the intranet. when I run my tests on the webserver machine MACHINE_A (it does triple-duty as test runner, webserver and db host) and export the results they export reasonably quickly as, obviously, the directory they are exporting to lies on that actual machine.

But when I run those tests on other machines and they have to export to the webserver directory on MACHINE_A they take a whole lot longer as they have to export across the network.

Just one example of why a test run could take longer on another machine.