Until yesterday test was running fine with the below highlighted test criteria in azure pipeline.
After Updating the test agent test are not getting filtered from the execution plan.
Execution plan looks like below. From which I am trying to run Menu5_US and its child test items. But the other groups also getting executed. Until yesterday only the specified group in the testFiltercriteria was getting executed in azure pipeline.
I've found a working workaround, by using the previous older TestAdapter version 1.15069.139, till the fix will be available.
I share it with you since this could temporarily resolve the issue also for you and let it possible to not block your test automation execution.
You can specify it as follows within your release pipeline, writing the complete working version on the task after InstallTestCompleteAdapter:
- task: InstallTestCompleteAdapter@1.15069.139
displayName: "TestComplete test adapter installer"
continueOnError: false
inputs:
installExecutor: false
updateExecutor: false
With this configuration, next VSTest tasks will work as expected.
Surely, when SmartBear or Microsoft will provide the fix, this workaround will need to be reverted to get the latest TestAdapter version.
Champion Level 3simonaferrara
Hi,
And thank you for reminding about this solution. It did not work a year or so ago when we had the same problem with Test Adapter, but now it works (at least for us). Probably, Azure fixed something that did not work previously...
Cheers,
/Alex
==========
Can you tell if we can do something like in azure classic pipelines
right after i posted my last message about the zoom call i had with the smartbear tech guy, he replied with a workaround solution for me and my azure classic pipelines. i've already implemented it on a couple of my VM's and verified that it does work. it is a way to use the previous, stable, working version of the test adapter which is in the zip file of the link below. hope this works for all of you. here is his email to me. Let me know if you have any questions.
Developers have reproduced and have been working on the issue. They will try to set up and implement a potential roll back strategy if there solutions don't work. Should have an update on Monday.
In the meantime, I actually had luck with successful pipelines using the following test adapter below:
It's the old one so it works fine. Do the following same steps again, but with this test adapter and see if you still have issues:
* Extract the zipped folder (may need to extract it in your desktop)
* the three dlls you need will be under "InstallTestCompleteAdapterTask\Modules"
* In your pipeline, disable the "TestComplete test adapter" step.
* In your VSTest step, look for "Path to custom test adapter" text field
* Copy the full path of the "InstallTestCompleteAdapterTask\Modules" to the textfield
They just release a newer version still not working
Champion Level 3Those that are impacted, I suggest you open a support ticket if not done already. So that the issue is escalated and made priority!
As a result of opening a case about this with Smartbear, I had a zoom call with one of their techs late yesterday afternoon. He recorded our session as I showed him an example of one of our pipelines that is no longer working as expected since the release of the new TestComplete test adapter. I first showed him how the execution plan in the TestComplete project suite has the three groups in the project and then showed the step in our pipeline that has the Group~<third group name> which is supposed to only run the test cases in that specific group. After starting the pipeline I showed him on the VM where it runs that it was ignoring the Group~<third group name> filter and was running the first test case from the first group name listed in the execution plan.
He tried to recreate the problem as he shared his screen with me but was unable to reproduce this issue on his end. Nevertheless, he saw it on my side and will be updating me with any developments. Unfortunately, the folks who are apparently going to be the ones working on this are in Europe so it clearly won't be until sometime today or early next week before I hear anything back. I did suggest that Smartbear should rollback to the previously stable, working release of the test adapter but I have not gotten a reply to that suggestion so far. If I hear anything I will be sure to post it to this group chat.
This workaround has indeed worked for us, Simona. Thank you.
I’ve got the same issue, from yesterday 17/2 in the morning.
The testFiltercriteria property of VSTest task seems not handled anymore; the tests are executed from the project’s first test item, without filtering by using this property.
Please quickly help to resolve the issue, since it highly impact our automatic tests execution.
If you have old version copy those TestAdapter files (it should be in Work folder of the devops agent) on C drive and in VSTest path to custom updaters can be the path. Below is the working example:
From my observation,
This issue just happens when test logs contain TYPE: WARNING. For Messages and Errors, it works properly.
Workaround solution: Add an event handler for OnLogWarning. In this solution, we must accept the trade-off that all warning logs in the test report will be hidden.
Configuration:
Implementation:
Please note that once the official fix is applied, these code changes must be reverted. This solution is just a temporary fix to avoid the blocking of CI/CD
Champion Level 3Is using the tilde character correct? It might not be supported on that particular version of the plugin.
AS per test complete documentation which I am referring is using ~ is correct to filter the groups.Run TestComplete Tests as Part of a Pipeline | TestComplete Documentation
Until yesterday the same filter expression was working as expected.
Champion Level 3Got it. ~ (tilde) represents a "contains" operator, meaning the filter selects all test cases where the Group field contains "Menu5_US".
Check if anyone else is having the same issue at https://github.com/microsoft/azure-pipelines-agent/issues . Otherwise you might have to create a bug report with https://github.com/microsoft/azure-pipelines-agent
We are also facing the same issue; The issue seems be due the Test Adapter update TestComplete test adapter - Visual Studio Marketplace. For the time being reverting to old version of Test Adapter worked. Have also created ticket Case #00694491
Can you share the steps how to use old version of Test Adapter. Can't able to get it to work.
