cancel
Showing results for 
Search instead for 
Did you mean: 

Ordering of Tests in Azure Pipeline

SOLVED
Highlighted
Contributor

Ordering of Tests in Azure Pipeline

Hello,  I have a bunch of tests which are running OK from an Azure Pipeline.

The testcases for the tests are all included in a Test Suite within a Test Plan and work as I want in the order outlined in the TestSuite; however, I would really like to split the tests into different Test Suites in the Test Plan, when I have tried this I lose control over the order in which the tests are run.

For example, if my first Test Suite has two steps:  Start, and Login in.  My second Test Suite runs a load of tests on the application, and a third Test Suite closes the application.  What happens is the first Test Case in the first Test Suite runs fine, then the first Test Case in the last Test Suite (which closes the application), then the Test Cases in the second Test Suite run against the application which has been closed.

I believe I have set things up correctly, as, Like I said, If I order the Test Cases within one Test Suite everything works OK.

Does anyone know if it is possible to ensure all the Test Cases within a Test Suite are run before the another one runs, and that Test Suites are run in a particular order? 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Moderator

Re: Ordering of Tests in Azure Pipeline

Hi @DanH,

 

The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

 

Julia Bernikova
SmartBear Customer Care Product Lead - TestComplete

View solution in original post

4 REPLIES 4
Highlighted
Community Manager

Re: Ordering of Tests in Azure Pipeline

Hi @DanH,

 

Thanks for posting your question. Could you please clarify the task a bit?

Are you setting up the order inside TestComplete? Could you please post several screenshots demonstrating the current set up? This should help the community better understand the current situation.

---------
Tanya Gorbunova
SmartBear Community Manager

Learn SmartBear products in a fun and easy way and prove your knowledge!
>>Participate in the TechCorner Challenge today
Highlighted
Contributor

Re: Ordering of Tests in Azure Pipeline

Hi Tanya,

 

Thanks for getting back.

The problem I have is the ordering of the tests when run using a Test Plan in an Azure Pipe Line:  There's no problem running them from a Test Complete project.

 

I have a number of Test Complete tests, which I have linked to Test Cases.  The test cases are added to a Test Plan, which is then run in an Azure Pipeline. These tests log on the a test application, import various data, enter some transactions and then close the application.  If I have the Test Cases in one Test Suite in the test plan, arranged in the correct order, they will run fine.  If I split the tests cases into different suites, covering different functionalities, the order the tests are run in breaks down.

 

In the attached screen shot, I have the tests arranged in the Test Suite "Start Prime and Log In to Regression Test data".  If I run just this Test Suite in the Pipeline, everything works.  If I include the Test Suites "Import Ledgers For Accounts", "Cribsheet Advanced Test Cases" and "Close Down" in the Pipeline (obviously getting rid of any Test Cases which would be duplicated from "Start Prime and Log In To Regression Test data"), the first test cases of the first Test Suite runs, starting the application, followed by the first test case of the last Test Suite, which closes the application, so all of the tests in the middle fail.  It's not a major problem as I can get round it by simply having all my Test Cases in a single Test Suite.  I was just wondering if it is possible to split the functionality into different Test Suites, and still maintain the order I want.  I hope that makes sense.

 

Dan

 

 

 

Highlighted
Moderator

Re: Ordering of Tests in Azure Pipeline

Hi @DanH,

 

The TestComplete test adapter has no control over the order in which Azure DevOps chooses to run the test suites, so the issue has to do with the Azure DevOps specifics. It might help if you add several VsTest tasks to the pipeline, one for each test suite - this way, they should be executed sequentially. If this setup gives the same result, I suggest you post a question to the Azure DevOps Developer Community.

 

Julia Bernikova
SmartBear Customer Care Product Lead - TestComplete

View solution in original post

Highlighted
Contributor

Re: Ordering of Tests in Azure Pipeline

Thanks Julia,

 

I thought it might be something like that

 

Dan

New Here?
Join us and watch the welcome video:
Announcements
Join TechCorner Challenge!
Want a fun and easy way to learn TestComplete? Try solving weekly TechCorner challenges and get into the Leaderboard!


Challenge Status

Compare images using the Region Checkpoint

See replies!

Converting UTC TimeDate in an Excel file

See replies!

Compare HTML table with Excel file and correct data in Excel file

Participate!

How to execute remote test and obtain results via Test Runner REST API

Participate!
Top Kudoed Authors