cancel
Showing results for 
Search instead for 
Did you mean: 

Execution Plan - Count - Stops on Failure

bschwass
Occasional Contributor

Execution Plan - Count - Stops on Failure

In the Execution Plan there is a column called "Count". My understanding of this column is it will cause the same test to run that many times while the plan is being executed. I have found that if one of those test runs were to fail, no more of those repeat runs will be done, and the execution will move onto the next test, even if "On Error" = "Continue running". 

 

We are currently in the process of investigating some flaky tests and it would be really handy if we could get say 10 failed results for the same test in an overnight run so that we can compare them. I don't see any settings that would make counts continue even after a failed test, but it would be really nice if there was one. 

 

The only workaround I've been able to find is just adding the same test to the Execution Plan multiple times, which is a bit cumbersome. 

Any ideas on a setting that I'm missing? Is there a feature request somewhere that I could +1?

6 REPLIES 6
tphillips
Community Leader

Re: Execution Plan - Count - Stops on Failure

I managed to reproduce your situation on my end.

I set the flaky test's On Error property to "Continue running" in the execution plan then it keeps going through the rest of the repetitions, however it also doesn't stop the current test. Anything after the Error message in that current item is still executed. If you are relying on the call to Log.Error to stop the current item, it looks like it won't do what you want.

I think TestComplete is doing exactly what it's told, so it's not necessarily a bug. TestComplete is just doing "Run this item until it fails, or until it's been run <count> times."

 

Maybe it would be worth creating a feature request for an option for the On Error property to have a new option for stopping the current iteration of the test and moving onto the next.


--------------------
QA Systems Architect
Marsha_R
Community Hero

Re: Execution Plan - Count - Stops on Failure

hkim5
Staff

Re: Execution Plan - Count - Stops on Failure

hkim5_0-1614976827483.png

is this not similar to the the desired outcome? if you were to raise that rerun # to be 10, then if you were to have flaky tests in your execution plan, this setting would in turn rerun that test 10 times.

 

if you have used this setting, and if it does not suffice, could you elaborate just a bit more about the desired outcome? I'm afraid im misunderstanding the final goal here.

Best,
Justin Kim
bschwass
Occasional Contributor

Re: Execution Plan - Count - Stops on Failure

That will rerun tests if they fail, but I want to run the test 10 times no matter what. 

AlexKaras
Community Hero

Re: Execution Plan - Count - Stops on Failure

Hi,

 

I want to run the test 10 times no matter what.

'Count' column of Execution Plan?

 

Regards,
  /Alex [Community Hero]
____
[Community Heroes] are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Posts made by [Community Heroes]
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
The [Community Hero] signature is used with permission by SmartBear Software.
https://community.smartbear.com/t5/custom/page/page-id/hall-of-fame
================================
bschwass
Occasional Contributor

Re: Execution Plan - Count - Stops on Failure

Alex, no that will stop running if one of the iterations fails. 

New Here?
Join us and watch the welcome video:
Announcements
Top Kudoed Authors