Re-running failed test cases

[nitinn]

Hello,

I am running a set of 15 tests.I want to re-run the failed test cases at the end of test execution.

Is there anyway through which I can identify the failed test cases and then re-run them at the end of execution.

Thank you

[tristaanogre]

Short answer: no.  There is no built in functionality in TestComplete to indicate that, if a test case is failed, to re-run it.

In truth, this is a feature/functionality of Test Management software like QA Complete or Jenkins... to identify failed test cases and then, with a set of clicks or whatever, re-run them.  TestComplete executes the tests and reports on them, other apps do the management of what needs to be re-run.

Long answer: If you write enough code, you can do this.  You would need to, somehow, write your tests in such a way that, if they "fail" (and you would need to clearly define a "failure" in code so as to make sure you properly identify it), you would somehow mark the test to be rerun (put the KeywordTest identifier or script routine reference into some sort of Array) and then write code to go through that list of tests and re-execute them.  It can be done.  As I'm fond of saying, you can automate ANYTHING given time, resource, energy and skill.  

[nitinn]

@tristaanogre

I am afraid that doesn't help me too much.

Isn't there any way through which I can just move the failed test items on the test items page to a new folder and then re-execute the new group again?

[baxatob]

Check this thread - https://community.smartbear.com/t5/TestComplete-General-Discussions/Run-failed-Tests-again/m-p/16070... - there is some workaround by @karkadil

[tristaanogre]

Sure... manually... but not in code and not automatically as part of a test run.  I'm sorry the answer doesn't help much... but that's the answer I have.  Again, TestComplete executes tests... MANAGING what gets executed and when it gets executed is what a Test Management tool is for... and TestComplete is not a Test Management tool.

As for moving test items around....

One of the ways I like to explain how a "test run" works in TestComplete is that, when you start a test run, there's a "compile" process that happens at the beginning that initializes the "stuff" of the test run... stuff that is loaded into memory BEFORE the tests start executing that control and dictate the rules of execution (stop on error, iteration count, test item dependencies, etc).   Part of that "compile" is the list of Test Items to be executed.  These are "read only" during a test run because they are "compiled" as part of what you are executing.  There are a number of things like this that are "initialized" when a test run starts which is why certain options, configuration settings, etc., are "read-only" at Run Time in TestComplete.  So, no, you cannot "move" TestItems into separate folders via code during Run Time of your tests.  

[tristaanogre]

---

@baxatob wrote:

Check this thread - https://community.smartbear.com/t5/TestComplete-General-Discussions/Run-failed-Tests-again/m-p/16070... - there is some workaround by @karkadil

---

Yes.  This will work.

But note... it's not something done completely within TestComplete.  Both @karkadil@ and @maxtester in another thread utilize an external tool to build something to be executed.  In both cases, they needed to write code (event handlers, something internal to the test case itself, etc), to mark and store what test cases are "failed" and then pass that information off to something else to build a "retest" of some sort... In one case, there's a PowerShell script that executes... I didn't read the other one but I assume that there's something similar going on that alters/builds the batch file to be executed.  Either way... you'll need to write custom code to do it.

[Marsha_R]

@nitinn Just curious -  why rerun the test cases again immediately?  Do you think they will pass the second time?  Perhaps if we help you look at why they are failing the first time, we can help you catch those problems and you won't need the second pass.

[tristaanogre]

---

@Marsha_R wrote:

@nitinn Just curious -  why rerun the test cases again immediately?  Do you think they will pass the second time?  Perhaps if we help you look at why they are failing the first time, we can help you catch those problems and you won't need the second pass.

---

That's actually a good question in general...  This isn't the first question we've gotten here about automatically re-executing.  And it probably won't be the last.

As a guess (because I don't bother with an automatic re-run myself) is that this is kind of a "sanity" check.  A test case fails... was it a fluke? Let's check to make sure... that's the only reason I would do something like this.

But it is precisely this reason why I wouldn't... usually if it's a "fluke"... that means something in my code of the test in the first place was done incorrectly... something was off on the timing, there was an object identification that didn't happen in time, there was some pre-condition I didn't account for, etc.  So... rather than just re-running it... I want to know what failed, then FIX it, THEN re-run it.  A failure is a failure...  Whether it's the application that failed or the test, I want to know about all failures and handle them before I run the test again.

[karkadil]

---

@nitinn Just curious -  why rerun the test cases again immediately?   

---

I can answer this 🙂

In our project we have a lot of long tests which run on unstable environments with complicated architecture. It takes about 15 hours to run them all.

We schedule runs at night on several PCs. After the 1st run only 50% of the tests pass. Most of the issues are not related to the AUT functionality, but either environment problems, or unstable script issues. Sometimes we are able to fix our own mistakes, sometimes we are not.

Now, when on the next day we simply rerun all failed tests, 50% of them usually pass. Other may require some tiny fixes. Some of them may fail on one PC, but pass on another. So in general, it is much easier to implement automatic rerun of the failed tests than to try to fix all possible issues 🙂

[tristaanogre]

---

@karkadil wrote:

---

@nitinn Just curious -  why rerun the test cases again immediately?   

---

Now, when on the next day we simply rerun all failed tests, 50% of them usually pass. Other may require some tiny fixes. Some of them may fail on one PC, but pass on another. So in general, it is much easier to implement automatic rerun of the failed tests than to try to fix all possible issues 🙂

---

Question: You mention that "on the next day, we rerun all failed tests".  Is this an automatic function of your code in TestComplete that, after the test run completes, it kicks off another run with the failed tests as an unattended execution?  Or does one of your staff come in and run the batch file generated that contains the failed tests?