I wanted to know on which environment do you run your tests ?
I have some TestExecute floating licences which make me able to run my tests on several machines simultaneously.
I have 2 desktop machines which runs part of my tests perfectly. But I also have a desktop with enough RAM and disk space to run HyperV VMs. But I noticed that theses machines are capricious. The same tests will most of the time fail on this HyperV VM when it could pass without errors in a desktop machine.
Do you use virtualization ? What are the specifications of theses machines ?
I tried to put 2 VM which each 5Gb of RAM and 30% of CPU.
I also tried VM on ESX and it was a pain to run my tests on theses machines.
If you could share your knowledge about virtualization it would be nice.
Thanks in advance,
Well, the failures on your virtual machines... it would be interesting to know what errors EXACTLY you're getting. My guess, from my experience, is that they are timing... a button or link or form is expected but doesn't appear in time and you get an "object not found" error.
This has been one of the biggest pieces in our virtualization puzzle. VMs share resource with a number of other machines on the same hardware. So, if that hardware is being taxed, even if the virtual CPU and RAM are not, the virtual machine could be impacted in performance causing any number of related issues.
Generally speaking, we like using virtualized machines because we can basically roll out machines based upon a specific template and guarentee that we'll always have machines that have the same general environment ensuring that our automation runs consistently.
Yes, it's timing issues.
About environments, the physical machines I have a dedicated to tests. Thus I have a stable environment to ensure that the tests will run ok.
I wish I have enough desktop machines but I can't have a bunch of 5 of ones... So I wish I could set up one or 2 VM.
Still, generally speaking, depending upon how the VMs are configured (are they actually using a dedicated core of the host CPU and dedicated RAM as well? Or are they sharing resource with the other VMs? What about drive swapping, etc?) That's the trickiness of virtual environments, in my experience. I think, no matter what, they will almost ALWAYS run a bit slower than a dedicated desktop machine unless you have some really kick-a$$ hosting environment.
As far as I understand you cannot allocate dedicated CPU cores to Hyper V. And very few hypervisors allow you to do so...
For the moment, just one HyperV per machine (so for each machine 2 TestExecute running at the same time) seems to be a good compromise.