Basic actions taking significantly longer when run as part of ADO pipeline

Our nightly CI/CD ADO pipeline runs tests on a dedicated VM solely for running these tests. We have ~140 tests which run over a period of 6-7 hours. 

When we execute tests as part of the release pipeline via ADO, the tests take on average 20% longer than they do when they are run manually via TestExecute on the exact same machine. Regardless of whether we run just 1 or all ~140 tests through the pipeline, the result is the same. 

The tests are executed using the tscon disconnection method to run headless, and the same script is run before the comparisons so it isn't related to that. 

Investigating further into the details, the execution time difference seems to come from extremely basic actions. An action that would take 1 second, can often take 15 seconds when the test is run as part of the pipeline. 

Some examples of these actions are:

A click with no key pressed.
Tested object:
Aliases.Application.MainForm.buttonMenu.wItems.Item["Browse"].Items.Item["Search"]
(Sys.Process("Application").WinFormsObject("MainForm").WinFormsObject("buttonMenu").wItems.Item["Browse"].Items.Item["Search"])

A click at point (10, 10) with no key pressed.

'^[F4]' was sent as a keyboard input to the following window or control.

Selecting values in combo boxes takes up to 7 seconds.

Even just giving focus to Edge takes 10 seconds:

'The window got focus' -Tested object: Aliases.browser.BrowserWindow (Sys.Browser("edge").BrowserWindow(0))

I've tried searching the community for similar threads but have had no luck. 

Any suggestions or help would be greatly appreciated.