New test 'collection'

Before I get into details, I'd like to first present the business case for this idea:

 

I am implementing a new deployment/test pipeline that uses the ready-api-maven-plugin to execute SoapUI tests after our webapps have been deployed inside a Docker container. There are 2 test suites that I would like to execute: One that checks all of the reports in our application, and another that just checks that a filter component is working correctly. The ready-api-maven-plugin has a <testSuite> property, but you can only provide a single test suite -- meaning my only options to execute both of my desired test suites is to:

1. Perform 2 Maven builds, one running each test suite
2. Make a new 'Smoke Test' test suite combining all of the test cases from each suite

 

Solution 1 adds a ton of overhead -- because I am using a container, it would have to startup a container, execute one test suite, teardown the container, then do the same thing for my 2nd test suite.

 

Solution 2 is a bit better, but would make managing the test suite a bit cumbersome and redundant -- Between my 2 test suites there are about 30 test cases -- I would have to manually, one-by-one, add all 30 of those test cases as 'Run test case' steps to this new test suite. In addition, by doing this, I lose a bit of granularity in the JUnit reports that are generated because now tests from my reports and filter suites are all grouped into one generic 'Smoke-Test' suite. One of the advantages of reporting on a test-suite level is that I can see at a glance if my reporting or filter is broken by just seeing if that test suite passed or failed.

 

I see 3 potential solutions:

 

1) A new 'Test Collection' option which would allow us to create a collection of tests while preserving the test suite they belong to. I think this would be the most complex to implement but would provide a lot of value. This is similar to just adding all of the desired test cases into one test suite, but would be easier to manage because you wouldn't need to add every test case one-by-one, and would (hopefully) still be able to see results on a test-suite level in your JUnit reports. I think the ready-api-maven-plugin would also need to be updated to add a new collection property. Below is an example:

 

// Test suite/case structure
Reports
   R1
   R2
   R3
   R4
   R5
Filter
   F1
   F2
   F3
Validation
   V1
   V2
   V3
   V4
   V5
   V6
   V7
   V8

// New test collection
Collection 1
   Reports
      R1
      R3
      R4
   Filter
      F2
   Validation
      V1
      V2
      V3

 

2) Add a new 'Run TestSuite' test step -- identical to the 'Run TestCase' step but will execute all of the test cases within that suite

 

3) Allow for multiple test suites to be specified in the <testSuite> property of the ready-api-maven-plugin and execute them in the order they are specified -- obviously for this to work, you would have to leave <testCase> empty.