Better Support of Test Data Sets in Test Script
I'm going to describe a problem I'm having when utilizing Test Data sets.
Example:
Test Data Set - Bikes Types - 8 Total
# of Steps in Test Script - 4
Steps 1 and 2 utilize the Bike Types test data set.
Steps 3 and 4 do not.
When this test script is executed, the following is created.
8 sets of tests with 4 steps each.
Steps 1 and 2 are repeated 8 times, which is fine because I want them to repeat 8 times, each time with a different Bike Type value.
Steps 3 and 4 are also repeated 8 times. This is not an ideal situation since I only want Step 3 and 4 tested once. Because of this I end up having 14 additional tests created unnecessarily.
Ideally, when the Test Script is executed, Steps 1 and 2 are repeated enough times to exhaust N number of values in the Test Data set it's utilizing, Steps 3 and 4 are only printed once because they do not have a Test Data set associated with them.
The way our organization sets up a Zephyr Scale Test Case, we want to have all tests for one story included in one Test Script. Is it possible for test data sets to only be executed against Steps that include them? And not against every single step, regardless of if it is using a Test Data set or not?
Additionally, this gets even more complicated if you have multiple sets of test data.
Example:
Test Data Set 1 - Bikes Types - 8 Total
Test Data Set 2 - Paint Color - 4 Total
# of Steps in Test Script - 4
Steps 1 and 2 utilize only the Bike Types test data set.
Steps 3 and 4 utilize only the Paint Color test data set.
When the test script is executed, it creates 8 sets of tests.
Steps 1 and 2 are repeated 8 times, to exhaust the total number of values in the Bike Types test data set, this is expected and wanted.
Steps 3 and 4 are also repeated 8 times. This is not expected and unwanted. Unfortunately, sets 5-8 will repeat Steps 3 and 4 and will create nonsense tests, since they're basing the number of repetitions on Test Data Set 1, and not Test Data Set 2. i.e. once you reach the 5th set of tests, you're going to have a test created with something like "Count the number of bikes with PAINT COLOR" for steps 3 and 4, since there are only 4 values in the Paint Color data set. The way that Zephyr Scale prints out the test sets, it's expecting 8 values.
This is only an example. This gets exponentially more complicated the more test sets you utilize.
Additionally, if you attempt to create a Step that utilizes more than one set of test data, this falls apart very quickly, as it only prints the values sequentially instead of creating an actual test matrix. i.e. using the above examples, if I wanted a test step that was like "Count the number of {Bike Types} with {Paint Color}" I'm only going to get 8 tests created in the execution, and it will just print out ;
Bike Type 1, Paint Color 1
Bike Type 2, Paint Color 2
Bike Type 3, Paint Color 3
Bike Type 4, Paint Color 4
Bike Type 5, Paint Color X (expecting a 5th value that doesn't exist since inheriting number of rows from Bike Type data set.)
Bike Type 6, Paint Color X (expecting a 6th value that doesn't exist since inheriting number of rows from Bike Type data set.)
Bike Type 7, Paint Color X (expecting a 7th value that doesn't exist since inheriting number of rows from Bike Type data set.)
Bike Type 8, Paint Color X (expecting an 8th value that doesn't exist since inheriting number of rows from Bike Type data set.)
This isn't expected nor is it ideal. In a perfect world, the steps of the test script would be printed out like this instead.
Bike Type 1, Paint Color 1
Bike Type 1, Paint Color 2
Bike Type 1, Paint Color 3
Bike Type 1, Paint Color 4
Bike Type 2, Paint Color 1
Bike Type 2, Paint Color 2
Bike Type 2, Paint Color 3
Bike Type 2, Paint Color 4
etc etc.
My expectation was that a test matrix would be created utilizing all combinations of the test data sets, not the way it's currently being created.
Some feedback/advice on this would be great. We had certain assumptions on how "smart" test data sets would be utilized in regards to test scripts/test steps, and so far support around test data sets seems to be extremely limited to only super basic uses.
