Forum Discussion

DanH's avatar
DanH
Contributor
5 years ago

Ordering of Tests in Azure Pipeline

Hello,  I have a bunch of tests which are running OK from an Azure Pipeline. The testcases for the tests are all included in a Test Suite within a Test Plan and work as I want in the order outlined ...
  • JuliaBernikova's avatar
    JuliaBernikova
    5 years ago

    Hi DanH,

     

    The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

     

TanyaYatskovska's avatar
TanyaYatskovska
SmartBear Alumni (Retired)

Hi DanH,

 

Thanks for posting your question. Could you please clarify the task a bit?

Are you setting up the order inside TestComplete? Could you please post several screenshots demonstrating the current set up? This should help the community better understand the current situation.

DanH's avatar
DanH
Contributor
5 years ago

Hi Tanya,

 

Thanks for getting back.

The problem I have is the ordering of the tests when run using a Test Plan in an Azure Pipe Line:  There's no problem running them from a Test Complete project.

 

I have a number of Test Complete tests, which I have linked to Test Cases.  The test cases are added to a Test Plan, which is then run in an Azure Pipeline. These tests log on the a test application, import various data, enter some transactions and then close the application.  If I have the Test Cases in one Test Suite in the test plan, arranged in the correct order, they will run fine.  If I split the tests cases into different suites, covering different functionalities, the order the tests are run in breaks down.

 

In the attached screen shot, I have the tests arranged in the Test Suite "Start Prime and Log In to Regression Test data".  If I run just this Test Suite in the Pipeline, everything works.  If I include the Test Suites "Import Ledgers For Accounts", "Cribsheet Advanced Test Cases" and "Close Down" in the Pipeline (obviously getting rid of any Test Cases which would be duplicated from "Start Prime and Log In To Regression Test data"), the first test cases of the first Test Suite runs, starting the application, followed by the first test case of the last Test Suite, which closes the application, so all of the tests in the middle fail.  It's not a major problem as I can get round it by simply having all my Test Cases in a single Test Suite.  I was just wondering if it is possible to split the functionality into different Test Suites, and still maintain the order I want.  I hope that makes sense.

 

Dan

 

 

 

  • JuliaBernikova's avatar
    JuliaBernikova
    SmartBear Alumni (Retired)
    5 years ago

    Hi DanH,

     

    The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.