Meet new TestComplete - v.10.20

Wow.



OK. Plugins enabled/disabled makes a MASSIVE difference.



Test overnight took 4hrs 20minutes. Switched off all plugins except the 3 or 4 the test actually needs ...... now runs in 18.5 minutes.



That is an INSANE difference.



Which makes me think even more that:



a) The should all be OFF by default so you only switch on the parts you need.

b) Their settings should be stored at project level. If I run several projects on the same VM, I now need to either change the plugin settings manually between each run or have all the stuff switched on to cover all runs. Neither way strikes me as very efficient.

c) Every time you upgrade the software, you need to remember to revert the plugin setting back to how they were pre-update.



At least I have the performance back now ....

Colin,



Thank you a lot for sharing your findings!

I suspected that extensions should slow-down tests run speed, but have never hit such big time differences like you did. Maybe this is because I never used some technologies supported by TestComplete and always turned off the relevant extensions every time when I upgraded TestComplete.



Considering your results I would like to vote for (Tanya, your attention, please):

a) All extensions should be enabled when TestComplete is installed a-new;

b) With no test project opened the state of extensions should be stored as the default project properties;

c) Extensions state should be a per-project setting and stored in the main project file (I don't think at the moment that extensions state should be different if the project is opened in the networked mode);

d) If TestComplete is in the version upgrade mode and installs a new extension, this extension should be enabled in the default project properties. Obviously, this new extension should be available from all already existing projects, but I am not sure whether it should be enabled or not in them.

I would like to see this as well:

b) With no test project opened the state of extensions should be stored as the default project properties;

c) Extensions state should be a per-project setting and stored in the main project file (I don't think at the moment that extensions state should be different if the project is opened in the networked mode);



I would like all new extensions disabled by default.

My practice is that, if it's not broken, don't touch it.

That is I would prefer my existing projects to maintain their current list of installed extensions without new ones popping up.

I'm still having major performance issues with TestExecute.



I raised this thread: http://smartbear.com/forums/f81/fp3/t88833/huge-performance-hit-using-testexecute-10-10/



And opened a support ticket. But that didn't come to much. I was asked to send in my entire project which I can't do for commercial reasons.



So I rolled everything back to v10.0 and Chrome (which seems to be the worst affected) to v31. However, Chrome v31 did not seem to be very stable and would crash frequently. However, after tweaking the project settings as far as possible, I did get it running at a good speed. But with the constant Chrome crashes, the test was not reliable.



Finding older Chrome installers is a nightmare. So I saw you had released v10.20 so I decided to try that with Chrome v34.



And the performance is now worse than ever.



Here are some timings:



TE = Test Ecexute

C = Chrome



TE v10.0 / C32 - untweaked project settings - Runtime = 1hr

TE v10.10 / C34 - untweaked project settings - Runtime = 2+ hrs

Rolled back

TE v10.0 / C31 - tweaked project settings - Runtime = 17 mins (but with crashes in Chrome)

TE v10.20 / C34 - tweaked project settings - Runtime = 4 hrs 20 mins !!!!!!!



So performance has gone way, way, downhill again.



Nothing changed between the last two runs listed except for updating TE and Chrome. The VM running all this is running Windows Server 2008 R2 64 bit. Xeon E5-2670 CPU. 2gb of RAM.



My initial thought was 2gb of RAM is not enough. So I left a perfmon trace running and RAM is only topping out at about 50%/60% peak used during the run.



As far as I can tell, TestExecute after v10.0 and Chrome v33/v34 are unusable.



My best option looks like reverting back to TE v10.0 (where I don't have any of these issues) but finding an installer for an older version of Chrome seems to be virtually impossible.



If anyone can point me to a stable v31/v32 install. That would be a HUGE help. The installer I have which does not appear to be stable is v31.0.1650.63. I've tried FileHippo but they don't actually host the older installers. It just directs you back to the Chrome site where the newest version is installed. Personally, I hate the way Chrome don't seem to make older installer available, but we appear to be stuck with it.



For the moment, I'm going back to trawling to try and find a stable Chrome v32 installer ....

Think I may have jumped the gun.



I thought all the enabled plugins were at project level. They're not. They an application setting.



So when I updated to v10.20, it switched on all the plugins I had switched off.



These seem to make a huge difference to performance.



Prior to using TestComplete, I used QTPro. Your approach to plugins and theirs is the opposite. With TC, everything is switched on by default. With QTPro, everything is off and you add in the plugins your test needs.



Not convinced having them all switched on by default is the best idea when they make this much difference to performance.



Or maybe the plugin settings could be stored at project level?



I'm re-running my 4hrs 20mins test now with nothing else changed except the plugin settings. I'll post back to see what difference they made but I think it's likely to be huge ...



And although this helps, it still doesn't explain the performance drop-off I saw when going from v10.0 to v10.10 & v10.20.



I didn't change any plugin settings when going from v10.0 to v10.10 and the test time did more than double. I only got the performance back by shutting down plugins.



So maybe there was a heavy duty plugin introduced at v10.10 which is hitting performance?



I didn't install any of the mobile testing modules when I updated ....

Agreed.



I have to admit, I prefer the QTPro approach of add things as you need them.



(I also like that QTPro can load object libraries - aka name maps - on the fly at run-time. And you can easily have multiple, centralised, shared, object libraries .... hint hint!)



When it starts making this much difference to the performance of the application (I'm still convinced the AMOUNT of difference the plugins make has increased dramatically since v10.10) having them all on by default does not seem like the way to go.



But don't worry! Apart from plugin handling and name/GUI maps, I prefer pretty much everything else in TC over QTPro!

In regard to the performance I have found it helpful to use the process filter as well under Tools>Current Project Properties>Open Applications>Process Filter.

This will minimize the impact certain plugins/built in methods have on performance.

Yep.



I already had only the relevant apps set up in there.



It was all the unused plugins causing the massive performance dent ....

> I would like all new extensions disabled by default.

The reason why I prefer to have them enabled in the default project properties is because for the disabled extension (or consider it as a feature) one must know about its existense and know where and how to enable it. Are all TestComplete users are curious enough to spend their time investigation a features that exist in the tool but are not used by them at the moment?

While when the feature is enabled by default, than the things appear to 'magically work out of the box' and an average user will not ask questions if this or that feature is supported or not.

When one will hit performance issues and will start disabling not needed extensions, this will mean that he already know what he is doing and there is a chance that he will remember that some feature is supported but is disabled in the given project.