Being Able to Pause a Test Running in TestExecute

[lphilippe_macau]

The idea is quite simple, being able to pause a test that is running in TestExecute. I know that you're supposed to be debugging in Testcomplete and that it's so people don't get licenses of TestExecute for development, but being able to pause a tests would be a big plus as some of our tests are very long to run and it would be nice to see if there is a problem on the environment without disturbing tests that are running. 

I don't think this would be hard to implement from the SmartBear team and it would really help our day to day. All of our QC Devs already have a licencse for TestComplete, but we would really need this to be more productive as we will currently wait for a test to be done to do validations on environments. 

Thanks

L-P

[leandropoblet]

I'm having this issue now on our application. Instead of starting maximized (as last user's option to be more precise) it randomly resets the value to default.

It's so random that I logged it as bug many times and it could never be reproduced by devs.

It also happens only on Win7 which is the only one running TE.

So the ideal solution for me would be pausing the execution, remote debugging of our app, and continue once it's been analyzed.

I really don't care about debugging TC scripts. It's just another useful tool to find sneaky bugs.

[Dmitry_Nikolaev]

Hello,

I would like to answer to the comment by leandropoblet.

I think that resolving this issue by pausing script execution is not the best solution as it won't work when you run nightly tests. If you are facing this situation, a better approach is to insert logic that will make sure that the application window is maximized to your test.

If the fact that the application is not maximized after it is started is an issue, I still do not see a reason to pause the test execution. This seems to be a bug and the test should be halted until the bug is resolved. Or you can handle this situation in your test, register it somehow and then just maximize the application window to continue test execution.

Dmitry Nikolaev

TestComplete Customer Care Team Lead

[leandropoblet]

Hi Dmitry,

This is just a very particular case used only as a sample of one way in which pausing is an advantage.

It's rather obvious it can't be used when running overnight, but as a user I always want to be able to do exactly what I want not just what the solution designer thought I was going to need or do with the product.

I've done what you're suggesting time ago.

Thanks,

Leandro

[Dmitry_Nikolaev]

 Hello, Community!

TestExecute is just a straightforward way to playback tests without any means to modify or debug them as this is the TestComplete functionality. We do not see any good reasons to move TestExecute functionality toward TestComplete and so generally this request does not seem to be something that we can implement.

Therefore, I would like to ask everyone if there are any actual use cases for this functionality that cannot be resolved with TestComplete or using other means.

Thanks in advance!

[lphilippe_macau]

@Dmitry_Nikolaev. I posted this a long time ago and of course there are ways around this. But it's was a productivity issue. We run our tests on virtual machines, they should all be the same, but Windows being Windows, they are not, so debugging environments is something that has to be done. When you want to run a diagnostic on an environment in the middle of a regression test that lasts hours, you either lose your test run or wait... Hitting a pause button would be so simple I can't believe you think it's not an issue. 

Also, all our tests are designed to reset the environment to it's original state (even the tested application). So if you find a bug and want to see details about the environment at at a certain point, you can't pause the test to do so. You have to go install TestComplete on those machines or try to rerun the steps manually which can be very long. 

Another one, is when random messages appear for things like Windows updates, Java updates, Adobe updates or any other random crap that can happen. You want to be able to dismiss these and clear out any open windows so that the test run can continue. Again, you either stop the test and start over you have the tests fail because of overlapping stuff. Normally not such of an issue, but when it's the last line of tests on release dates, this is time consuming on an already overbooked date. 

I can find more, but I'm not the one that deals daily with this anymore. 

[Dmitry_Nikolaev]

Hello Friends.

We are trying the understand the underlying issue for the current request, and I have another question related to it.

How do you think, will remote debugging of a test running with TestExecute on another machine solve the problem?

[lphilippe_macau]

@Dmitry_Nikolaev , I'm sorry but you do not seem to understand how your product works. We're not trying to debug with TestExecute, we're trying to pause the test without having to restart the whole test run. We have more or less 40 VM's running test excecute through various jobs on different products at all times. Some of the test benches that we built in Test Complete are over 40 hours to run. So if you can't pause TestExecute and a problem happens on the VM, every test just starts failing. But sometimes someone can simply go in and fix the issue on the VM and voila everything would be fine for the the remaining tests. This is a big productivity issue. Also, most problems are not related to the way the tests are built, but more related to the environment. Planned downtimes for file servers, application servers, related resources or anything similar are into this category.

So here is a common example:

• - We schedule a test run to execute on an application at 8:00 pm
• - There is a planned maintenance on a file server at 11:00 pm for 30 minutes
• - We would like to be able to pause Test Execute on all VM's during the maintenance window. 
• - If we cannot do this we loose 3 hours of tests per VM, so this could be up to 120 hours lost simply because everything will fail during and probably after the maintenance window. 

[JuliaBernikova]

Hi @lphilippe_macau,

Thank you for providing an example of a situation where you'd need to pause a test.

As I understand, you'd like the TestExecute indicator to include the "Pause" button just like the one available in TestComplete. So, in case of some planned downtime, someone from your team would need to manually pause the execution and resume it when the environment is available again, is that correct?

Given that some of the test suites take more than 40 hours to finish and maintenance usually takes place outside of working hours, pausing the execution manually might not be the most convenient approach. Do you use any test management/build automation tools (like Jenkins, Azure DevOps, etc.)? Maybe, is it possible to use them to automate the test execution and overcome the issues related to planned maintenance? For example, you can schedule some of the tests to run before the server downtime and the rest of them to be executed after it. It's also possible to automatically rerun failed tests.

Also, you mentioned that sometimes someone can simply connect to the machine and fix the problem that makes the tests fail. However, to do that, you first need to understand that something is wrong - see the error log. So, the execution is already stopped by the time it's clear that something went wrong on playback, and it doesn't look like pausing a test can help in this case. Could you please provide more details on this situation? How do you determine that there's a problem and manual "intervention" is required?

[BenoitB]

As for me I encountered several times where i've said "ERRRKKK need to restart everything !!" because no pause on TestExecute.

Sometimes when you need to do lot of manual settings of environment before launching long lasting test and you miss a little thing or it has changed somewhere a little but that implies that you KNOW the test will fail then a pause execution would be so helpful.

And to manage the case of @lphilippe_macau  do you think the QA team know in advance when the the maintenance occurs ? I don't think so. Moreover on large system you have intelligent maintenance that can occurs at any time when the system detect the need of (self healing provisioning system).

About people connecting to machine and fix ... Yes of course, the target is to know why it's going wrong but the goal is to deliver test results within time so pausing will help to achieve the goal and AFTER they will investigate on erro to get rid of it.

Last thing is i think it's a big mistake assuming that TE is always under a full devops environment with tons of tools. TE is also used in fully functional environment (end user environment) without these tools. Lot of my customers have only a TE running on their system without any other tools (because of budget, because of system constraints, because of ..).

That was my 5 cents 😉