Forum Discussion

bschwass's avatar
bschwass
Occasional Contributor
4 years ago

Execution Plan - Count - Stops on Failure

In the Execution Plan there is a column called "Count". My understanding of this column is it will cause the same test to run that many times while the plan is being executed. I have found that if one of those test runs were to fail, no more of those repeat runs will be done, and the execution will move onto the next test, even if "On Error" = "Continue running". 

 

We are currently in the process of investigating some flaky tests and it would be really handy if we could get say 10 failed results for the same test in an overnight run so that we can compare them. I don't see any settings that would make counts continue even after a failed test, but it would be really nice if there was one. 

 

The only workaround I've been able to find is just adding the same test to the Execution Plan multiple times, which is a bit cumbersome. 

Any ideas on a setting that I'm missing? Is there a feature request somewhere that I could +1?

count
Execution Plan
Repeat
Test Results
  • hkim5's avatar
    hkim5
    Staff
    4 years ago

    is this not similar to the the desired outcome? if you were to raise that rerun # to be 10, then if you were to have flaky tests in your execution plan, this setting would in turn rerun that test 10 times.

     

    if you have used this setting, and if it does not suffice, could you elaborate just a bit more about the desired outcome? I'm afraid im misunderstanding the final goal here.

    • bschwass's avatar
      bschwass
      Occasional Contributor
      4 years ago

      That will rerun tests if they fail, but I want to run the test 10 times no matter what. 

      • AlexKaras's avatar
        AlexKaras
        Champion Level 3
        4 years ago

        Hi,

         

        > I want to run the test 10 times no matter what.

        'Count' column of Execution Plan?

         

  • tphillips's avatar
    tphillips
    Frequent Contributor
    4 years ago

    I managed to reproduce your situation on my end.

    I set the flaky test's On Error property to "Continue running" in the execution plan then it keeps going through the rest of the repetitions, however it also doesn't stop the current test. Anything after the Error message in that current item is still executed. If you are relying on the call to Log.Error to stop the current item, it looks like it won't do what you want.

    I think TestComplete is doing exactly what it's told, so it's not necessarily a bug. TestComplete is just doing "Run this item until it fails, or until it's been run <count> times."

     

    Maybe it would be worth creating a feature request for an option for the On Error property to have a new option for stopping the current iteration of the test and moving onto the next.