Showing results for 
Search instead for 
Did you mean: 

Data driven testing

New Contributor

Data driven testing

My co-worker posted this question at:

Since then, I found out we had to login with the account provided when we purchased SoapUI.

The question is:

We have a very complex SOAP web service.  It provides CRUD operations to several objects in our server database.
We have ~approximately 12 objects which we refer to as 'primary objects' that can be passed into these CRUD operations.  A few examples are contacts and organizations. 

Each primary object can have multiple sub-objects and multiple elements/attributes.

We have an operation in our WSDL that will return metadata which describes each object and element exposed in our web service.  An example of the metadata for an object would be the object name, what it inherits from, a list elements/attributes.  The metadata for an element/attribute contains the attribute name, data type, (base types or complex types), nullability, data range and usage for each CRUD operation, etc. 

In total, we have 60+ objects and 500 + fields.
Some of our test cases include attempting a CRUD operation on a primary object  with valid and invalid elements based on the usage from the metadata. 

The question we have is how can we drive our testing based on this metadata?
We can live with not having metadata retrieved at runtime by dumping the metadata before executing the test suite and using that dump as a data source.

How can we do multiple iterations through the data source which will drive composition of the SOAP requests?

An example the metadata will contact a contact the contact may contain 40-50 fields.  Let's say 5 of those fields are required on create, 5 of the fields are not allowed on create and the rest are allowed but not required on create.  We want to execute a test request where only the 5 required fields are specified, and verify we receive a valid response.  We also want to test we can specify all required and allowed fields and receive a valid response.  Then, we will want to test error conditions where we build 5 requests, each one omitting one of the required fields and verify that we receive a SOAP fault.

Additionally, we want to build 5 requests that specify the required fields and each one will specify a not allowed field and verify that we receive a SOAP fault.

We would want to do similar testing with the read, update, and destroy operations.  We would also want to use a similar concept for validating field data ranges, etc.
We believe the only effective way to accomplish this and have a maintainable test suite is through a data-driven approach where the data is from metadata.

Is this possible in SOAPUI.?

Re: Data driven testing


well.. it is definitely possible but would (probably) require quite alot of scripting to generate the requests/assertions/etc from your metadata.. can you give some indication on xml format of this metadata so I can try to help you forward?



Did my reply answer your question? Give Kudos or Accept it as a Solution to help others. ⬇️⬇️⬇️
New Contributor

Re: Data driven testing

This is roughly what the XML schema should looks like

<xs:element name="MetaDataAttribute" type="MetaDataAttribute"/>
<xs:complexType name="MetaDataAttribute">
<xs:element name="CustomField" type="xs:boolean" minOccurs="1" maxOccurs="1"/>
<xs:element name="DataType" type="rng:DataTypeEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="Deprecated" type="xs:boolean" minOccurs="1" maxOccurs="1"/>
<xs:element name="Description" type="xs:string" minOccurs="1" maxOccurs="1"/>
<xs:element name="KeyType" type="MetaDataKeyTypeEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="ListType" type="MetaDataListTypeEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="MaxLength" type="xs:int" minOccurs="1" maxOccurs="1"/>
<xs:element name="MaxListSize" type="xs:int" minOccurs="1" maxOccurs="1"/>
<xs:element name="MaxValue" type="xs:int" minOccurs="1" maxOccurs="1"/>
<xs:element name="MinValue" type="xs:int" minOccurs="1" maxOccurs="1"/>
<xs:element name="Name" type="xs:string" minOccurs="1" maxOccurs="1"/>
<xs:element name="Nullable" type="xs:boolean" minOccurs="1" maxOccurs="1"/>
<xs:element name="Pattern" type="xs:string" minOccurs="1" maxOccurs="1"/>
<xs:element name="UsageOnCreate" type="MetaDataUsageEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="UsageOnDestroy" type="MetaDataUsageEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="UsageOnGet" type="MetaDataUsageEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="UsageOnUpdate" type="MetaDataUsageEnum" minOccurs="1" maxOccurs="1"/>
<xs:element name="UsedAsName" type="xs:boolean" minOccurs="1" maxOccurs="1"/>
<xs:element name="MetaDataClass" type="MetaDataClass"/>
<xs:complexType name="MetaDataClass">
<xs:element name="Attributes" type="MetaDataAttribute" minOccurs="0" maxOccurs="unbounded"/>
<xs:element name="CanCreate" type="xs:boolean" minOccurs="0" maxOccurs="1"/>
<xs:element name="CanDestroy" type="xs:boolean" minOccurs="0" maxOccurs="1"/>
<xs:element name="CanGet" type="xs:boolean" minOccurs="0" maxOccurs="1"/>
<xs:element name="CanUpdate" type="xs:boolean" minOccurs="0" maxOccurs="1"/>
<xs:element name="Class" type="rng:RNObjectType" minOccurs="0" maxOccurs="1"/>
<xs:element name="DerivedFrom" type="rng:RNObjectType" minOccurs="0" maxOccurs="1"/>
<xs:element name="Relationships" type="MetaDataRelationship" minOccurs="0" maxOccurs="unbounded"/>
<xs:simpleType name="MetaDataUsageEnum">
<xs:restriction base="xs:string">
<xs:enumeration value="NOT_ALLOWED"/>
<xs:enumeration value="ALLOWED"/>
<xs:enumeration value="IGNORED"/>
<xs:enumeration value="REQUIRED"/>

I have also included the Usage enum. It relates directly to the CRUD operations in the WSDL. So if UsageOnCreate is REQUIRED, this means the field is required on the create operation and the server will throw a soap fault if it is not set.

We will also want to drive a significant amount of testing based off of the Relationships data. The MetaDataRelationship complexType basically describes a relationship between the current class and another class. I figured I did not need to specify the details for this since whatever we do for the CRUD operations would likely apply to the other testing.

The meta-data is based on a CSV file that is checked into source control. So we have the option of using the CSV file or using the meta-data at runtime.

Another design option we were considering was to write a simple app (maybe in C#/.NET) that would take the meta-data (either from the CSV or runtime) and emit a bunch of data source files that could be used in data source loop steps. Would this be a better design? It seems like it would be easier to maintain and easier to track changes since we could source control the data source files.

Re: Data driven testing

I do think that the solution you are suggesting would be the best one... Good luck with that, and please let us know how it goes, would love to see how a project of that size and scope works with SoapUI.


Did my reply answer your question? Give Kudos or Accept it as a Solution to help others. ⬇️⬇️⬇️
New Here?
Join us and watch the welcome video:
Top Kudoed Authors