Azure Pipeline - Visual Studio Test - Test execute fails to run (but was working yesterday)

[SlickRick]

Well, another day another question. 😄

Yesterday my build was working fine and my tests were passing on a given project.

Today it seems like my Visual Studio Test  suddenly fails with the following message:

"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs /logger:trx "/TestAdapterPath:\"D:\agent\_work\24\s\""

Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)

Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Failed to get a list of tests from the "D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs" file due to the following error: Cannot start process because a file name has not been provided.

No test is available in D:\agent\_work\24\s\Src\XXX.UITests_TC\XXX.UITests_TC.pjs. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

No changes were made to my project and no change to the pipeline itself. It is a bit strange.

I tried running testexecute directly on the machine in question and it is running fine.

I was wondering if anyone got this issue and could point me in the right direction. I find the logs to be not very verbose, i don't even see how TestExecute is called... 😕 

[AlexKaras]

@SlickRick :

Hi,

I believe that I can confirm this problem. Support case #00560527 has been created for it.

To check if your case corresponds to mine, can you check the output provided to pipeline log by TestComplete Test Adapter step for the passed and failed pipeline execution? I am interested in the version value. For me, pipeline worked with Adapter version 1.15040.421 and started to fail when adapter updated itself to version 1.15053.9.

@D0UG :

Doug, could you please draw Support's attention to this case? The thing that makes it to be a critical showstopper is that it seems to be not possible to rollback adapter to some previous version and/or prohibit automatic updates. (Well, at least our DevOps failed to find how to do this. I will really appreciate it if someone describes here how to do this if this is possible.)

[SlickRick]

Yes! this is exactly my case...

The version that worked is indeed:

Version : 1.15040.421

and  the failing one is:

Version : 1.15053.9

I would have never though of checking this one... 

[AlexKaras]

Just for Support's cross-reference: the thread with exactly same problem https://community.smartbear.com/t5/TestComplete-Questions/TestComplete-Test-Adapter-v15-53-9-issues-...

[AlexKaras]

@rwood_cert :

Thank you for your suggestion.

https://stackoverflow.com/questions/72946728/yaml-specifying-a-fully-qualified-azure-task-version

and

https://stackoverflow.com/questions/72946728/yaml-specifying-a-fully-qualified-azure-task-version

sounded promising, but failed for me with this error on pipeline validation or if started:

"A task is missing. The pipeline references a task called 'InstallTestCompleteAdapter'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 1.15040.421, job '[skipped]', step ''.)"

[jrsbr]

We're getting the same error.  Also attempted to run TestExecute and TestComplete manually on the test agents and was unable to, it just hung.  The Azure DevOps pipleline logging was identical.

[rwood_cert]

We're having to work on getting one of our projects back into active testing, as we can't await this fix.

Have got this running in a YAML pipeline, example below, believe you have to specify the full task name/reference

            - task: SmartBearSoftware.install-testcomplete-adapter-task.install-testcomplete-adapter-task.InstallTestCompleteAdapter@1.15040.421
              displayName: 'TestComplete test adapter installer'
              inputs:
                installExecutor: false
                updateExecutor: false
                accessKey: '$(TestExecuteAccessKey)'
                logsLevel: 0

[AlexKaras]

@rwood_cert :

Thank you for the information and screenshot.

Alas, it did not work for me with the same error about missed task.

Our pipeline is in YAML. Did you do anything else except specifying task as in your example? Do you have administrative permissions in DevOps/for your pipeline?
I noted that you have installExecutor property set to false... Did you uninstalled Adapter before running the pipeline or the task handled roll-back itself?

[rwood_cert]

As some background, hopefully the below helps, the YAML example further down is effectively what we're using as a workaround, to use the prior DevOps task version and latest TestExecute version.

We're running the pipeline on a Windows build agent pool (a VMSS in Azure), those VMs are built using a VM image with TestExecute preinstalled (v15.53.2 in this case), hence our installExecutor and updateExecutor being set to false.

This pool is setup in DevOps with the interactive tests option enabled, this option forces the need to preinstall TestExecute, as the build agent user runs unelevated so can't install/update - https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops#c...

Anonymised example of one of the stages in the pipeline, I've added comments where tasks removed or info randomised.

stages:
  - stage: SmokeTests
    displayName: 'Smoke Tests (PROJECTNAME)'
    jobs:
    - deployment: 'SmokeTests'
      environment: AutomatedTests
      variables:
      - group: AutomatedTest-PROJECTNAME-VariableGroup
      pool: 
        name: 'UITestPool'
      displayName: 'Smoke Tests (PROJECTNAME)'
      strategy:
        runOnce:
          deploy:
            steps:
            - checkout: self ## Checks out our TestComplete project/tests
            - task: DownloadBuildArtifacts@1
              displayName: 'Download Artifact 1'
              inputs:
                buildType: 'specific'
                project: '61b13bb3-6ba0-44a0-9d67-753c0a63752b' ## GUID replaced in this example
                pipeline: '1' ## The Build definition ID where published artifact is
                buildVersionToDownload: 'latest'
                downloadType: 'specific'
                itemPattern: '**'
                downloadPath: '$(System.DefaultWorkingDirectory)\_Licensing'

            ## Download build artifact 2, same as task above

			## Powershell task modifying Licensing installer name

            - task: SmartBearSoftware.install-testcomplete-adapter-task.install-testcomplete-adapter-task.InstallTestCompleteAdapter@1.15040.421
              displayName: 'TestComplete test adapter installer'
              inputs:
                installExecutor: false
                updateExecutor: false
                accessKey: '$(TestExecuteAccessKey)'
                logsLevel: 0

            - task: ms-autotest.screen-resolution-utility-task.screen-resolution-utlity-task.ScreenResolutionUtility@1
              displayName: 'Set Screen Resolution'
              inputs:
                displaySettings: specific
                width: 1920
                height: 1080

            ## Installation and Licensing tests, same task as below, running different tests

            - task: VSTest@2
              displayName: 'Smoke Tests'
              inputs:
                testAssemblyVer2: '**\<PROJECTNAME>.pjs'
                testFiltercriteria: 'Group=SmokeTests'
                testRunTitle: 'Smoke Tests (PROJECTNAME)'
                timeoutInMinutes: 60

[D0UG]

@AlexKaras I've reached out to our support team. I'll report back when I have answers.