Azure DevOps Pipelines - Running “Headless” Tests

One of the most common requests I get from TestComplete customers who run their tests from Azure DevOps pipelines is: “How can I execute my tests on remote Virtual Machines (VMs) with self-hosted agents without needing to maintain an active terminal session?”.

Note: This document does not detail setting up the Azure DevOps pipeline or the self-hosted Azure agents. That process can be found in the TestComplete documentation here: Integration With Azure DevOps and Team Foundation Server via TestComplete Test Adapter | TestComplete Documentation

There are some details in the TestComplete documentation on how to accomplish this type of configuration, but I’ll go through the full process and how to configure this solution with two options. That documentation can be found online: Disconnecting From Remote Desktop While Running Automated Tests | TestComplete Documentation

The first option is to set up multiple virtual machines (VMs). You start by logging into one VM via Remote Desktop Protocol (RDP). From there, you connect to other Tester VMs running Microsoft Agents. After issuing a command to release the RDP session back to the Admin user, the Tester VMs remain active, allowing TestExecute to run tests triggered by the pipeline.

The second option is to use a single, high-performance VM to host all remote sessions and agents. Here, multiple Tester sessions run on the same VM, each with its own agent handling pipeline requests. Each session runs its own TestExecute instance as long as enough floating licenses are available. The Admin session connects to the VM and then accesses each Tester session via RDP using the VM’s loopback address (e.g., 127.0.0.2). However, this method requires the VM to be part of a Microsoft Domain to support more than two simultaneous RDP sessions.

Here are the commands we’ll be taking advantage of in this operation.

To gracefully disconnect from an active RDP terminal session, converting it into a console session, open a Command Prompt (CMD) in the Admin user session and use the following:

                %windir%\System32\tscon.exe RDP-Tcp#NNN /dest:console

where RDP-Tcp#NNN is the ID of your current Remote Desktop session, for example, RDP-Tcp#5. You can see it in the Windows Task Manager on the Users tab, in the Session column. The Session column is hidden by default. To show it, right-click somewhere within the row that displays CPU, Memory, and so on, then choose Session in the opened context menu.

Our documentation also includes an easy-to-use batch file option that can be run as an Administrator to more easily disconnect from the remote Admin session.

Create a batch file with this code:

for /f "skip=1 tokens=3" %%s in ('query user %USERNAME%') do ( %windir%\System32\tscon.exe %%s /dest:console
)

1. Create a desktop shortcut to this file. To do this, right-click the batch file and select Send to > Desktop (create shortcut).
2. In the shortcut properties, click Advanced and select Run as administrator.
3. Double-click this shortcut on the remote computer (in the Remote Desktop Connection window) or call this batch file at the beginning of your tests (provided that the tests are run as an Administrator).

 

To gracefully reconnect to a remote console session, use the following command in an open CMD prompt:

                mstsc -v: servername /F -console

where the servername is the address of the remote VM hosting the Admin session.

Option 1: Multiple VMs in the testing network.

1. RDP connect to and set up the remote VMs to be used for the Tester Agents and have their self-hosted Microsoft agents active and listening for jobs, then disconnect the RDP sessions.
2. RDP connect to the VM to be used as the “Admin” session.
3. From the “Admin” VM, RDP connect to each Tester VM. (Connecting to more than 2 remote RDP sessions will require the VMs to be on a Domain with the appropriate settings configured to allow for more than 2 remote sessions.)
4. Your Admin system should look something like this image where I am connected to 2 of my Tester VMs with their agents listening for jobs:

5. Ensure that your Agents are active in the Azure DevOps Agent pool:

6. Disconnect from the Admin user session using the prepared batch file or the direct command with the RDP session ID:

7. Run the pipeline to validate that tests execute as expected:

8. Reconnect to the Admin VM to see test results from the running agents using the command:

mstsc -v: servername /F -console

 

Option 2: Using a Single Virtual Machine

1. RDP connect to the VM to be used for testing using the “Admin” account. I’m using my Tester01 account in this demonstration.
2. RDP connect to the other users on the same VM using the 127.0.0.2 loopback address.
3. Run the Microsoft self-hosted agents on these connected sessions.

4. Disconnect from the Admin user session using the prepared batch file or the direct command with the RDP session ID.

5. Prepare the pipeline. In my case, as I am not on a Domain so I am limited to ONE RDP session from my “Admin” user. I’m disabling the agent on Tester03 since we are using Tester02 for our pipeline.
6. Execute the pipeline and validate successful test executions:

7. Reconnect to the “Admin” session to make changes to the configuration as needed:

mstsc -v: servername /F -console

 

Conclusion

In conclusion, these two methods provide testing options from Azure DevOps using VMs without requiring a dedicated terminal session from your development system. They still require that session, but the solutions provide options to workaround that “directly monitored” connection.

These solutions also give you the security of running them on VMs in your environment; easily secured, controlled, and isolated by your firewall and security requirements.

Full demonstration video link:

Azure DevOps Pipelines - Remote "Headless" Testing