Forum Discussion

DanH's avatar
DanH
Contributor
5 years ago

Ordering of Tests in Azure Pipeline

Hello,  I have a bunch of tests which are running OK from an Azure Pipeline.

The testcases for the tests are all included in a Test Suite within a Test Plan and work as I want in the order outlined in the TestSuite; however, I would really like to split the tests into different Test Suites in the Test Plan, when I have tried this I lose control over the order in which the tests are run.

For example, if my first Test Suite has two steps:  Start, and Login in.  My second Test Suite runs a load of tests on the application, and a third Test Suite closes the application.  What happens is the first Test Case in the first Test Suite runs fine, then the first Test Case in the last Test Suite (which closes the application), then the Test Cases in the second Test Suite run against the application which has been closed.

I believe I have set things up correctly, as, Like I said, If I order the Test Cases within one Test Suite everything works OK.

Does anyone know if it is possible to ensure all the Test Cases within a Test Suite are run before the another one runs, and that Test Suites are run in a particular order? 

  • JuliaBernikova's avatar
    JuliaBernikova
    5 years ago

    Hi DanH,

     

    The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

     

  • TanyaYatskovska's avatar
    TanyaYatskovska
    SmartBear Alumni (Retired)
    5 years ago

    Hi DanH,

     

    Thanks for posting your question. Could you please clarify the task a bit?

    Are you setting up the order inside TestComplete? Could you please post several screenshots demonstrating the current set up? This should help the community better understand the current situation.

    • DanH's avatar
      DanH
      Contributor
      5 years ago

      Hi Tanya,

       

      Thanks for getting back.

      The problem I have is the ordering of the tests when run using a Test Plan in an Azure Pipe Line:  There's no problem running them from a Test Complete project.

       

      I have a number of Test Complete tests, which I have linked to Test Cases.  The test cases are added to a Test Plan, which is then run in an Azure Pipeline. These tests log on the a test application, import various data, enter some transactions and then close the application.  If I have the Test Cases in one Test Suite in the test plan, arranged in the correct order, they will run fine.  If I split the tests cases into different suites, covering different functionalities, the order the tests are run in breaks down.

       

      In the attached screen shot, I have the tests arranged in the Test Suite "Start Prime and Log In to Regression Test data".  If I run just this Test Suite in the Pipeline, everything works.  If I include the Test Suites "Import Ledgers For Accounts", "Cribsheet Advanced Test Cases" and "Close Down" in the Pipeline (obviously getting rid of any Test Cases which would be duplicated from "Start Prime and Log In To Regression Test data"), the first test cases of the first Test Suite runs, starting the application, followed by the first test case of the last Test Suite, which closes the application, so all of the tests in the middle fail.  It's not a major problem as I can get round it by simply having all my Test Cases in a single Test Suite.  I was just wondering if it is possible to split the functionality into different Test Suites, and still maintain the order I want.  I hope that makes sense.

       

      Dan

       

       

       

      • JuliaBernikova's avatar
        JuliaBernikova
        SmartBear Alumni (Retired)
        5 years ago

        Hi DanH,

         

        The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

         

  • DanH's avatar
    DanH
    Contributor
    5 years ago

    Thanks Julia,

     

    I thought it might be something like that

     

    Dan