In the Execution Plan there is a column called "Count". My understanding of this column is it will cause the same test to run that many times while the plan is being executed. I have found that if one of those test runs were to fail, no more of those repeat runs will be done, and the execution will move onto the next test, even if "On Error" = "Continue running".
We are currently in the process of investigating some flaky tests and it would be really handy if we could get say 10 failed results for the same test in an overnight run so that we can compare them. I don't see any settings that would make counts continue even after a failed test, but it would be really nice if there was one.
The only workaround I've been able to find is just adding the same test to the Execution Plan multiple times, which is a bit cumbersome.
Any ideas on a setting that I'm missing? Is there a feature request somewhere that I could +1?
I set the flaky test's On Error property to "Continue running" in the execution plan then it keeps going through the rest of the repetitions, however it also doesn't stop the current test. Anything after the Error message in that current item is still executed. If you are relying on the call to Log.Error to stop the current item, it looks like it won't do what you want.
I think TestComplete is doing exactly what it's told, so it's not necessarily a bug. TestComplete is just doing "Run this item until it fails, or until it's been run <count> times."
Maybe it would be worth creating a feature request for an option for the On Error property to have a new option for stopping the current iteration of the test and moving onto the next.
is this not similar to the the desired outcome? if you were to raise that rerun # to be 10, then if you were to have flaky tests in your execution plan, this setting would in turn rerun that test 10 times.
if you have used this setting, and if it does not suffice, could you elaborate just a bit more about the desired outcome? I'm afraid im misunderstanding the final goal here.
Regards, /Alex [Community Hero] ____ [Community Heroes] are not employed by SmartBear Software but are just volunteers who have some experience with the tools by SmartBear Software and a desire to help others. Posts made by [Community Heroes] may differ from the official policies of SmartBear Software and should be treated as the own private opinion of their authors and under no circumstances as an official answer from SmartBear Software. The [Community Hero] signature is used with permission by SmartBear Software. https://community.smartbear.com/t5/custom/page/page-id/hall-of-fame ================================