TFS - Distributed Testing on a Remote Machine w TestExecute

[Southpaw]

Hello,

   I'm hoping to get some help with my build definition tasks in TFS (2015) to get TC tests to run using Test Execute on remote hosts. I have successfully created a Machine Group in TFS and no failures using the Visual Studio Test Agent Deployment task. I assume this is one of the steps I need to use for remote host testing out of TFS. But the Visual Studio Test task is still trying to execute the test(s) locally. See screen shot for my task steps. I think i need another step but just dont know which one. 

Thanks

[Southpaw]

Hello - 

   Ok I have found some progress. Please Note we are using TFS 2015. The Build Def Tasks Im using currently is:

1) Visual Studio Build

2) Windows Machine File Copy

3) Visual Studio Test Agent Deployment

4) Run Functional Tests

Ive made a connection to our remote host pool. I am able to copy TesComplete project files to a network share. VS Test Agent deployment and configuration succeeds. But the Run Functional Tests begins but ultimately fails with the error 

DistributedTests: Error : No tests were discovered from the specified test sources

This is odd because I have all of the TC Project Files copied to the network share and all tasks map to this share. So something is wrong and I cant figure it out. ARE THERE ANY SmartBear Customers that actually use TC Integrated with VS and subsequently use TFS to run tests on distributed machines???

This is beyond frustrating because there is a mountain of info to set up a distributed test run for Selenium, etc.. and I find nothing regarding TC. SmartBear docs say nothing about this approach. 

Thanks.

[JuliaBernikova]

Hello,

Thank you for posting your question here. The good news is that the approach you use is the one recommended in the TestComplete documentation (see the "Run TestComplete Tests as Part of Build Pipeline (Team Builds)" article). So, the steps you used in the build definition are absolutely correct.

As for the "No tests were discovered from the specified test sources" error: you mentioned that the TestComplete project files had been copied to the specified location. However, the "Run Functional Tests" task can only run Visual Studio tests, not TestComplete projects. So, the agent is looking not for TestComplete project files (.pjs or .mds), but for TestComplete Visual Studio test files that reference actual TestComplete projects. These TestComplete Visual Studio tests files are the .tcTest or .tc12Test files. They are added to the Visual Studio solution and work as mediators between Visual Studio and TestComplete. The "Run Functional Tests" task tries to find these files by the mask specified in the "Test Assembly" field ("**\*.tc*Test"). So, first of all, please make sure that the location (\\ussdvucsd107\downloads\TestComplete) does indeed contain the .tcTest or .tc12Test files the agent is looking for.

If the location only contains TestComplete projects, please follow the "2. Prepare Visual Studio Test Projects for Running TestComplete Tests (Run TestComplete Tests as Pa..." and "3. Define a Build Process (Run TestComplete Tests as Part of Build Pipeline)" articles to make sure the solution is prepared and copied to the network location correctly.

If the location does actually contain .tcTest or .tc12Test files, please make sure that the TestComplete integration package is installed on the test host. To see the entire list of requirements, please refer to "Run TestComplete Tests as Part of Build Pipeline - Requirements".

Please also note that this approach is only recommended if you want to fully incorporate TestComplete tests into your TFS workflow and run them as usual Visual Studio tests. The approach might indeed be somewhat confusing as it involves a lot of steps to "trick" Visual Studio into thinking TestComplete tests are Visual Studio tests. But if your goal is to simply run TestComplete tests from TFS without any special reporting about the results, you can follow much simpler steps. Just copy the TestComplete projects to the test host, compose the command line necessary to run them (see "TestComplete Command Line") and execute this command line by using the "Command line" TFS task. Remember to make the TFS test agent interactive - that's required no matter the approach.

[RRostovtsev]

Hi Southpaw,

There should not be any difference between distributed run of TC tests and standard tests, except that TestComplete test adapter should be installed on remote machines. You should also understand that *.tcTest (or *.tc12Test) files are actually considered as tests by TFS, not *.pjs.

As far as I understand, this error

No tests were discovered from the specified test sources

can occur in two cases:

1) TestComplete test adapter is not installed on a remote machine (Have you installed Visual Studio integration or Visual Studio Test Agent integration on your remote machines?)

2) Test files are not found in the "Drop location folder" (Are you sure that *.tc*Test files are present in your \\ussdvucsd107 share?)

Please also note that Microsoft marked "Run Functional Tests" step as deprecated and recommends to use "Visual Studio Test" step instead in next versions of TFS. And it is a good practice to store tests in a source control.

Best regards,

Roman

[Southpaw]

When you say TestComplete test adapter - what exactly is this? Ive installed TestExecute on the remote machine that I am testing out these features on. I also have the VS Integration installed on the remote machine. 

I'm aware of the deprecated task. I do not know what else to use as this task allows me to enter a machine group that contains the remote machine for this effort. The Visual Studio Test task doesnt offer that and when I run my build it attempts to run the test(s) on the build machine. 

If you know the task when on TFS2015 to get the test to actually kick off on the remote machine(s) please let me know. 

[RRostovtsev]

Hi Southpaw,

Don't worry about TestComplete test adapter. I meant that you need Visual Studio integration or Visual Studio Test Agent integration. Since you have installed one of them on the remote computer, probably the reason is that .tcTest files are absent on the remote machine.

Regarding Visual Studio Test task, next versions of TFS provide with a new more flexible release pipiline which allows to distribute tests among machines, and it uses this task.

[TanyaYatskovska]

Thanks for the help, Julia, Roman!

@Southpaw, was the issue resolved? Please give Kudos to replies and accept the best answer as a solution to say thanks to guys.

[Southpaw]

Hi - No the issue is not resolved. My build def runs without error. However, the test is not executed. I have a support ticket opened for this same issue that has yet to be resolved. No one has explained how this - having test execute or test complete actually runs the test(s) - when TFS Build Def's are used to kick off remote test execution. On a side note, I also have a ticket open regarding using TestComplete for the remote test execution using a Network Suite. I am unable to verify the host machine. Ive had destop engineering, AT&T engineer's (they manage our clients firewall) and the security team all try and assist me in why I keep getting a verification failure that says server not started. All the requirements are met. So from my side, this feature does not work. Sorry, I wish this all was resolved because Ive put a mountain of time into proving this feature works for my management team. 

[TanyaYatskovska]

Hi @Southpaw,

It's sad to hear this. Have you contacted the TestComplete Support team regarding this question? If you haven't, I suggest that you reach out to them:

https://support.smartbear.com/message/?prod=TestComplete

[Southpaw]

Hi @TanyaYatskovska  I am currently working w a couple of SmartBear support engineers on both issues. I believe the Network Suite not verifying host is going to development. The other is still being communicated between myself and suuport. I appreciate your attentiveness to this issue. 

Chris