Forum Discussion

DanH's avatar
DanH
Contributor
5 years ago

Ordering of Tests in Azure Pipeline

Hello,  I have a bunch of tests which are running OK from an Azure Pipeline. The testcases for the tests are all included in a Test Suite within a Test Plan and work as I want in the order outlined ...
  • JuliaBernikova's avatar
    JuliaBernikova
    5 years ago

    Hi DanH,

     

    The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

     

DanH's avatar
DanH
Contributor

Hi Tanya,

 

Thanks for getting back.

The problem I have is the ordering of the tests when run using a Test Plan in an Azure Pipe Line:  There's no problem running them from a Test Complete project.

 

I have a number of Test Complete tests, which I have linked to Test Cases.  The test cases are added to a Test Plan, which is then run in an Azure Pipeline. These tests log on the a test application, import various data, enter some transactions and then close the application.  If I have the Test Cases in one Test Suite in the test plan, arranged in the correct order, they will run fine.  If I split the tests cases into different suites, covering different functionalities, the order the tests are run in breaks down.

 

In the attached screen shot, I have the tests arranged in the Test Suite "Start Prime and Log In to Regression Test data".  If I run just this Test Suite in the Pipeline, everything works.  If I include the Test Suites "Import Ledgers For Accounts", "Cribsheet Advanced Test Cases" and "Close Down" in the Pipeline (obviously getting rid of any Test Cases which would be duplicated from "Start Prime and Log In To Regression Test data"), the first test cases of the first Test Suite runs, starting the application, followed by the first test case of the last Test Suite, which closes the application, so all of the tests in the middle fail.  It's not a major problem as I can get round it by simply having all my Test Cases in a single Test Suite.  I was just wondering if it is possible to split the functionality into different Test Suites, and still maintain the order I want.  I hope that makes sense.

 

Dan

 

 

 

JuliaBernikova's avatar
JuliaBernikova
SmartBear Alumni (Retired)
5 years ago

Hi DanH,

 

The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.