Forum Discussion

KJ1's avatar
KJ1
New Contributor
2 months ago
Solved

Azure DevOps TestComplete Re-run failed test cases does not work

Hi

I am running TestComplete tests in Azure using the TestComplete test adapter. Visual Studio Test task has an option to re-run failed tests.

During rerun, It tries to find the test case with vstest.console.exe /TestCaseFilter:"(FullyQualifiedName=ProjectSuite.TestCaseName)". Is it possible to use instead (~ and testcase name) FullyQualifiedName~TestCaseName. We are not using fully qualified name in test complete.

It gives the error -> No test matches the given testcase filter `(FullyQualifiedName=ProjectSuite.TestCaseName)`.

How can we make rerun work? There are certain test cases which sometimes fail in initial run because of some intermittent issues so rerunning is required.

 

4 Replies

  • Hassan_Ballan's avatar
    Hassan_Ballan
    Icon for Champion Level 3 rankChampion Level 3
    2 months ago

    This is a known issue when using Azure DevOps' “rerun failed tests” feature with the TestComplete Test Adapter. Azure filters reruns with:

    /TestCaseFilter:"FullyQualifiedName=ProjectSuite.TestItem:Project\\TestCaseName"

    But this format often doesn't align with how TestComplete identifies test cases, so reruns fail with "no tests found." The adapter doesn't currently support using ~ (contains) or partial matching in FullyQualifiedName.

    Workarounds:

    1. Custom rerun logic:
      Parse the first run’s results (.trx or JUnit) to get failed tests, then re-run them in a separate step using filters the adapter supports (e.g. tags or test item names).
    2. Use TestComplete’s built-in retry:
      Configure retries directly in the Execution Plan (On error → rerun current item).
    3. Check with SmartBear:
      While you're likely on a recent version, SmartBear can confirm if recent adapter updates improve rerun behavior or support more flexible filtering.

    Docs:
    Test Adapter Setup
    Tag-Based Filtering Setup

    🤖 AI-assisted response
    👍 Found it helpful? Click Like
    ✅ Issue resolved? Click Mark as Solution

  • KJ1's avatar
    KJ1
    New Contributor
    2 months ago

    I have already tried the first option. But problem merging the test result counts. If there are 4 test cases and 2 of them fails. Rerunning 2 failed tests gives total count of Run tests as 6 instead of 4. It doesn't merge the results of .trx files generated.

     

  • KJ1's avatar
    KJ1
    New Contributor
    2 months ago

    Also, since initial run failed but rerun was successful, it marks the task as failed. Though there is an option to continue on error. But this marks the pipeline as (either failed or partial successful). This is by design in Azure Devops if any task fails . If initial run fails but rerun is successful we will never get success result from pipeline.