New test 'collection'
Before I get into details, I'd like to first present the business case for this idea:
I am implementing a new deployment/test pipeline that uses the ready-api-maven-plugin to execute SoapUI tests after our webapps have been deployed inside a Docker container. There are 2 test suites that I would like to execute: One that checks all of the reports in our application, and another that just checks that a filter component is working correctly. The ready-api-maven-plugin has a <testSuite> property, but you can only provide a single test suite -- meaning my only options to execute both of my desired test suites is to:
- Perform 2 Maven builds, one running each test suite
- Make a new 'Smoke Test' test suite combining all of the test cases from each suite
Solution 1 adds a ton of overhead -- because I am using a container, it would have to startup a container, execute one test suite, teardown the container, then do the same thing for my 2nd test suite.
Solution 2 is a bit better, but would make managing the test suite a bit cumbersome and redundant -- Between my 2 test suites there are about 30 test cases -- I would have to manually, one-by-one, add all 30 of those test cases as 'Run test case' steps to this new test suite. In addition, by doing this, I lose a bit of granularity in the JUnit reports that are generated because now tests from my reports and filter suites are all grouped into one generic 'Smoke-Test' suite. One of the advantages of reporting on a test-suite level is that I can see at a glance if my reporting or filter is broken by just seeing if that test suite passed or failed.
I see 3 potential solutions:
1) A new 'Test Collection' option which would allow us to create a collection of tests while preserving the test suite they belong to. I think this would be the most complex to implement but would provide a lot of value. This is similar to just adding all of the desired test cases into one test suite, but would be easier to manage because you wouldn't need to add every test case one-by-one, and would (hopefully) still be able to see results on a test-suite level in your JUnit reports. I think the ready-api-maven-plugin would also need to be updated to add a new collection property. Below is an example:
// Test suite/case structure Reports R1 R2 R3 R4 R5 Filter F1 F2 F3 Validation V1 V2 V3 V4 V5 V6 V7 V8 // New test collection Collection 1 Reports R1 R3 R4 Filter F2 Validation V1 V2 V3
2) Add a new 'Run TestSuite' test step -- identical to the 'Run TestCase' step but will execute all of the test cases within that suite
3) Allow for multiple test suites to be specified in the <testSuite> property of the ready-api-maven-plugin and execute them in the order they are specified -- obviously for this to work, you would have to leave <testCase> empty.