Execution Plan - Count - Stops on Failure

[bschwass]

In the Execution Plan there is a column called "Count". My understanding of this column is it will cause the same test to run that many times while the plan is being executed. I have found that if one of those test runs were to fail, no more of those repeat runs will be done, and the execution will move onto the next test, even if "On Error" = "Continue running". 

We are currently in the process of investigating some flaky tests and it would be really handy if we could get say 10 failed results for the same test in an overnight run so that we can compare them. I don't see any settings that would make counts continue even after a failed test, but it would be really nice if there was one. 

The only workaround I've been able to find is just adding the same test to the Execution Plan multiple times, which is a bit cumbersome. 

Any ideas on a setting that I'm missing? Is there a feature request somewhere that I could +1?

[tphillips]

I managed to reproduce your situation on my end.

I set the flaky test's On Error property to "Continue running" in the execution plan then it keeps going through the rest of the repetitions, however it also doesn't stop the current test. Anything after the Error message in that current item is still executed. If you are relying on the call to Log.Error to stop the current item, it looks like it won't do what you want.

I think TestComplete is doing exactly what it's told, so it's not necessarily a bug. TestComplete is just doing "Run this item until it fails, or until it's been run <count> times."

Maybe it would be worth creating a feature request for an option for the On Error property to have a new option for stopping the current iteration of the test and moving onto the next.

[Marsha_R]

Here's where you can add a feature request:

https://community.smartbear.com/t5/TestComplete-Feature-Requests/idb-p/TestXCompleteFeatureRequests

[hkim5]

is this not similar to the the desired outcome? if you were to raise that rerun # to be 10, then if you were to have flaky tests in your execution plan, this setting would in turn rerun that test 10 times.

if you have used this setting, and if it does not suffice, could you elaborate just a bit more about the desired outcome? I'm afraid im misunderstanding the final goal here.

[bschwass]

That will rerun tests if they fail, but I want to run the test 10 times no matter what. 

[AlexKaras]

Hi,

> I want to run the test 10 times no matter what.

'Count' column of Execution Plan?

[bschwass]

Alex, no that will stop running if one of the iterations fails.