Ask a Question

Better Support of Test Data Sets in Test Script

thx1138
Visitor

Better Support of Test Data Sets in Test Script

I'm going to describe a problem I'm having when utilizing Test Data sets.

 

Example:

Test Data Set - Bikes Types - 8 Total

# of Steps in Test Script - 4

Steps 1 and 2 utilize the Bike Types test data set.

Steps 3 and 4 do not.

 

When this test script is executed, the following is created.

8 sets of tests with 4 steps each.

Steps 1 and 2 are repeated 8 times, which is fine because I want them to repeat 8 times, each time with a different Bike Type value.

Steps 3 and 4 are also repeated 8 times. This is not an ideal situation since I only want Step 3 and 4 tested once. Because of this I end up having 14 additional tests created unnecessarily.

Ideally, when the Test Script is executed, Steps 1 and 2 are repeated enough times to exhaust N number of values in the Test Data set it's utilizing, Steps 3 and 4 are only printed once because they do not have a Test Data set associated with them.  

 

The way our organization sets up a Zephyr Scale Test Case, we want to have all tests for one story included in one Test Script. Is it possible for test data sets to only be executed against Steps that include them? And not against every single step, regardless of if it is using a Test Data set or not?

Additionally, this gets even more complicated if you have multiple sets of test data.

Example:

Test Data Set 1 - Bikes Types - 8 Total

Test Data Set 2 - Paint Color - 4 Total

# of Steps in Test Script - 4

Steps 1 and 2 utilize only the Bike Types test data set.

Steps 3 and 4 utilize only the Paint Color test data set.

 

When the test script is executed, it creates 8 sets of tests.

Steps 1 and 2 are repeated 8 times, to exhaust the total number of values in the Bike Types test data set, this is expected and wanted. 

Steps 3 and 4 are also repeated 8 times. This is not expected and unwanted. Unfortunately, sets 5-8 will repeat Steps 3 and 4 and will create nonsense tests, since they're basing the number of repetitions on Test Data Set 1, and not Test Data Set 2. i.e. once you reach the 5th set of tests, you're going to have a test created with something like "Count the number of bikes with PAINT COLOR" for steps 3 and 4, since there are only 4 values in the Paint Color data set. The way that Zephyr Scale prints out the test sets, it's expecting 8 values.

This is only an example. This gets exponentially more complicated the more test sets you utilize.

 

Additionally, if you attempt to create a Step that utilizes more than one set of test data, this falls apart very quickly, as it only prints the values sequentially instead of creating an actual test matrix. i.e. using the above examples, if I wanted a test step that was like "Count the number of {Bike Types} with {Paint Color}"  I'm only going to get 8 tests created in the execution, and it will just print out ;

Bike Type 1, Paint Color 1

Bike Type 2, Paint Color 2

Bike Type 3, Paint Color 3

Bike Type 4, Paint Color 4

Bike Type 5, Paint Color X (expecting a 5th value that doesn't exist since inheriting number of rows from Bike Type data set.)

Bike Type 6, Paint Color X (expecting a 6th value that doesn't exist since inheriting number of rows from Bike Type data set.)

Bike Type 7, Paint Color X (expecting a 7th value that doesn't exist since inheriting number of rows from Bike Type data set.)

Bike Type 8, Paint Color X (expecting an 8th value that doesn't exist since inheriting number of rows from Bike Type data set.)

This isn't expected nor is it ideal. In a perfect world, the steps of the test script would be printed out like this instead.

Bike Type 1, Paint Color 1

Bike Type 1, Paint Color 2

Bike Type 1, Paint Color 3

Bike Type 1, Paint Color 4

Bike Type 2, Paint Color 1

Bike Type 2, Paint Color 2

Bike Type 2, Paint Color 3

Bike Type 2, Paint Color 4

etc etc.

My expectation was that a test matrix would be created utilizing all combinations of the test data sets, not the way it's currently being created. 

 

Some feedback/advice on this would be great. We had certain assumptions on how "smart" test data sets would be utilized in regards to test scripts/test steps, and so far support around test data sets seems to be extremely limited to only super basic uses. 

1 REPLY 1
MisterB
Community Hero

Re: Better Support of Test Data Sets in Test Script

Hi,

 

Your issue sounds similar to the one that I just commented on for Piwil.  It sounds like you need the DDT feature and the ability to connect to an external source so that you can add the level of complexity you need in something like Excel.  But it doesn't seem to be available as yet.

 

Data-Driven Testing (DDT) - Overview | Zephyr Scale Server/Data Center Documentation (smartbear.com)

 

It has been requested as a feature enhancement on the ideas board for both server and cloud.  I've posted a couple of links to those 'ideas' below (for the Server version) where you could add your vote if you wish (I've added mine).  Sorry that doesn't help you with your immediate requirement.

 

As a user, I can upload datasets via | Zephyr Scale Server/DC Ideas (aha.io)

 

As a user, I can import parameters & | Zephyr Scale Server/DC Ideas (aha.io)

 

Best, Andy

cancel
Showing results for 
Search instead for 
Did you mean: