testFiltercriteria not applied

Originally posted by rupali_sonawane on February 18, 2025

Until yesterday test was running fine with the below highlighted test criteria in azure pipeline.

After Updating the test agent test are not getting filtered from the execution plan.

Execution plan looks like below. From which I am trying to run Menu5_US and its child test items. But the other groups also getting executed. Until yesterday only the specified group in the testFiltercriteria was getting executed in azure pipeline.

Originally posted by rraghvani on February 18, 2025

Is using the tilde character correct? It might not be supported on that particular version of the plugin.

Originally posted by rupali_sonawane on February 18, 2025

AS per test complete documentation which I am referring is using ~ is correct to filter the groups.Run TestComplete Tests as Part of a Pipeline | TestComplete Documentation

Until yesterday the same filter expression was working as expected.

Originally posted by rraghvani on February 18, 2025

Got it. ~ (tilde) represents a “contains” operator, meaning the filter selects all test cases where the Group field contains “Menu5_US”.

Check if anyone else is having the same issue at https://github.com/microsoft/azure-pipelines-agent/issues . Otherwise you might have to create a bug report with https://github.com/microsoft/azure-pipelines-agent

Originally posted by NikhileshJ on February 18, 2025

We are also facing the same issue; The issue seems be due the Test Adapter update TestComplete test adapter - Visual Studio Marketplace. For the time being reverting to old version of Test Adapter worked. Have also created ticket Case #00694491

Originally posted by dhundley on February 18, 2025

i am seeing something similar. Starting yesterday, the completion status (Passed or Failed) are not appearing in the azure devops results log as the pipeline runs. I’ve also noticed that, in a pipeline that has multiple steps and each step is set up to run on a different VM that it is ignoring the Agent.Name equals in the Demands section and is running on a VM where it’s not supposed to causing it to use the wrong sets of login credentials for that test. I am having to disable all my pipelines until this is resolved. Can the version of the TestComplete test adapter that was released yesterday be rolled back to the prior release temporarily to see if that alleviates the issue?

Originally posted by Dbowerman1 on February 18, 2025

Following

Originally posted by scot1967 on February 18, 2025

Following too. :thinking:

Originally posted by NikhileshJ on February 18, 2025

Unfortunately, I do not see any option to cherry pick adapter version. It by default installs the faulty one. If you have some pipeline which have not been run with latest adapter, you can copy the old adapter folder from the work directory/s of that pipeline and use it. You will have to-

  1. Provide the hardcoded path of the adapter.
  2. In case you’re passing some command line parameters, then you will have to write some script to update Adapter options.json file as per your need.

After doing above, it works for me.

Originally posted by abdanish on February 18, 2025

Running into same issue following to

Originally posted by ashaik on February 18, 2025

Can you share the steps how to use old version of Test Adapter. Can’t able to get it to work.

Originally posted by Bruce1 on February 18, 2025

Major Impact, as we have 9 Test Executes running a few hundred tests!

Originally posted by bhapriravi on February 18, 2025

We are also badly impacted.

We use the following filters for tests from ADO based on the test selector: test plan. We provide Test Plan and Test Suite IDs to actually run the tests, as shown below. This stopped working since yesterday.

We need this fix ASAP.

task: VSTest@3

          displayName: regressiontest

          inputs:

            testSelector: testPlan

            testPlan: 33802

            testSuite: '237345'

            testConfiguration: 43

            vsTestVersion: toolsInstaller

          continueOnError: true

Originally posted by ButtonPusher on February 18, 2025

I’m having the same issue here. Been down for 2 days… 100’s of tests failing.

Originally posted by simonaferrara on February 19, 2025

I’ve got the same issue, from yesterday 17/2 in the morning.

The testFiltercriteria property of VSTest task seems not handled anymore; the tests are executed from the project’s first test item, without filtering by using this property.

Please quickly help to resolve the issue, since it highly impact our automatic tests execution.

Originally posted by rraghvani on February 19, 2025

Those that are impacted, I suggest you open a support ticket if not done already. So that the issue is escalated and made priority!

Originally posted by Chandrika_nag1 on February 19, 2025

Hi NikhileshJ , How are you hardcoding the older version? I am using ADO pipeline tasks where for this I am using “TestComplete test adapter installer” task, with this task how can i use older version of test adapter?

Originally posted by ashaik on February 19, 2025

If you have old version copy those TestAdapter files (it should be in Work folder of the devops agent) on C drive and in VSTest path to custom updaters can be the path. Below is the working example:

Originally posted by rupali_sonawane on February 19, 2025

Chandrika_nag1I tried adding version # in “TestComplete test adapter installer” task but it still installs the faulty one.

  • task: InstallTestCompleteAdapter@1
    displayName: “Installing Test Complete Test Adapter”
    inputs:
    accessKey: ‘d74f4d38-7eb6-459c-aa55-b1e318980316’
    version: ‘1.15069.139’

Originally posted by NikhileshJ on February 19, 2025

If you have the old version with you(if you’re lucky you can find it in old pipeline src folder), If you’re using VSTest@2 task, you can hard code it here

  • -task: VSTest@2
    displayName: ‘Start TestExecute runner’
    inputs:
    testAssemblyVer2: ‘project.pjs’
    testFiltercriteria: ‘Project=Test&Group~“to_research”’
    vsTestVersion: toolsInstaller
    pathtoCustomTestAdapters: Test AdapterPath(you need to copy the old version here)
    publishRunAttachments: false
    preferredExecutor: TE

I understand it is very specific fix and will work only if you somehow manage to have old Adapter version, still putting it out here if someone benefits from it.