cancel
Showing results for 
Search instead for 
Did you mean: 

Data Driven Testing?

msiadak
Community Hero

Data Driven Testing?

I'm splitting this off after accidentally hi-jacking another thread. Basically what it comes down to is I have to generate a lot of data in the course of testing the services presented to me as thoroughly as possible. Some of our services are significantly large and I end up with hundreds of test cases. The problem stems from the fact that these test cases often require unique data for visibility into issues and for constraints within the service itself. (UUID's, etc).

 

The problem comes in that the Add requests in these services can, in and of themselves, have hundreds of elements. One of my larger projects right now has 150~ requests (positive and NULL tests, where the null tests contain elements that can be empty or omitted completely). Couple that many requests with the fact that there are ~300 elements per request, you are talking about roughly 45,000 pieces of data that I need access to.

 

The problem comes in that, to properly test is that I do have to get nitty-gritty and verify data is different because I may have to turn around and update this, which can easily drive my number of test cases and property requests to nearly double.

 

In the other thread, it was recommended that I look at Data Driven testing and this would be done with Data Sources and Data sinks, etc. I've looked at this and have read up on it and it sounds like something I need to explore. Of course, nothing is ever that simple because I have to keep in mind my own constraints. Namely, however I set this up to test and work has to be easily reproduced by other people. The problem here is that I cannot just set up a database for this, as the other people running my test projects may not have the capability to do the same.


That leaves me with, as far as I can see, using DataSource and files. Is this really the best way to handle this? I could write all of the properties to files and then pull them back in. 


Am I missing something here? Am I doing something wrong? I'm relatively new to all of this, as I've been working with SoapUI for only a little over a year.

 

The problem at hand is I am currently storing all of my data in a Properties step. The more and more properties I have to store, the longer and longer it takes to save/open/close my SoapUI project. Currently, the example project I mentioned with 45,000 properties can take at least 4-5 minutes to open. If I open some of my other projects that are just as large, eventually SoapUI pegs at 100% CPU and 6+GB of memory. My current system has 32gb of memory and a good CPU, but gets bogged down under the strain of the sheer amount of properties.


Any and all advice/help is much appreciated. 




---

Click the Accept as Solution button if my answer has helped, and remember to give kudos where appropriate too!
11 REPLIES 11

Re: Data Driven Testing?

Would it be possible for you to use Excel files for your Data Sources and Data Sinks? The files could be shared or placed on a network drive for multiple testers to access.

Amber
msiadak
Community Hero

Re: Data Driven Testing?

From what I've seen I would have to import extra JAR files into SoapUI for this to work. The issue isn't other testers local to me running the project. The problem is there may be people using my tests that I would not necessarily have direct contact with if they run into issues.

 

I am trying to keep this as simple and small as possible. The projects I make need to be as stand-alone as possible, as not everyone will have the capability to add outside/3rd party jar files to SoapUI. 

 

I am facing what may be a lot of unique constraints.




---

Click the Accept as Solution button if my answer has helped, and remember to give kudos where appropriate too!
MFagerlind
Staff

Re: Data Driven Testing?

Are you using the Open Source version or Ready! API?

 

It should be possible to handle this kind of data using Data sources in Ready! API, but I don't really think the free OS version will cut it.

 

We've had some issues with very large projects with a huge number ofproperties but this has been mostly fixed now. I believe the fix was in 1.6; if not it will be in the upcoming 1.7 release.

 

Regards,

Manne, developer

msiadak
Community Hero

Re: Data Driven Testing?

I'm currently on Ready! API 1.6. I try to stay on the latest version as much as I can. I did notice an improvement with 1.6, but over time SoapUI NG Pro still bogs down and can take minutes to open a project of this size. Can you verify the fix for large amounts of properties being fixed in 1.6 versus 1.7? I'm curious to know.

 

I'm going to explore the Data Sources option and see where I can get with that with a proof of concept with my team. If this works we may be able to make this a requirement for when we deliver our SoapUI projects for testing verification to our partners. 




---

Click the Accept as Solution button if my answer has helped, and remember to give kudos where appropriate too!
MFagerlind
Staff

Re: Data Driven Testing?

I checked the merge date: the fix didn't make it into 1.6 unfortunately, but will be in 1.7. I'd be interested to hear whether the 1.7 performance is good enough for you.

 

Regards,

Manne

msiadak
Community Hero

Re: Data Driven Testing?

 I appreciate the heads up on that. I'll definitely be keeping my eyes peeled for 1.7 to see if that alleviates what I am seeing. In the mean time, I am going to be looking more into Data Source and what it can do for me. Thanks, everyone, for the help. It is really appreciated. I'll report in when 1.7 hits (any ETA?) with whether or not I see any performance gains. 




---

Click the Accept as Solution button if my answer has helped, and remember to give kudos where appropriate too!
MFagerlind
Staff

Re: Data Driven Testing?

According to the Product owners, the preliminary release date is April 19. I'll try to remember to post another reply in this thread when it's released, so you'll get an email notification if you subscribe to it.

 

Happy testing!

 

Manne

MFagerlind
Staff

Re: Data Driven Testing?

Just saw this in one of your earlier posts in this thread:

 

> From what I've seen I would have to import extra JAR files into SoapUI for this to work.

 

This is only true if you're running the Open source version of SoapUI. The Excel functionality works out of the box in Ready! API. If you're on the OS version then you're on your own in the wondrous realm of Groovy scripts, though. 🙂

 

Manne

 

 

msiadak
Community Hero

Re: Data Driven Testing?

Following up on the train of discussion.

 

Previous to 1.7.0 releasing, I ended up implementing the following methodology to store all of the many, varied properties I need access to:

  1. Generate data, write to datasink/local postgres database. 
  2. Set up datastore to access local postgres database.
  3. Built script assertion to manually step through datastore record(s).

 

The above worked for allowing me to test the four or five relatively large services without having ReadyAPI hang and/or crash on me.

 

 

Now that 1.7.0 is out and I've been able to install it, I've noticed that ReadyAPI is handling large projects / groovy scripts generating 50,000~ pieces of data a lot better. Unfortunately, opening said project can still take 30-60 seconds. So there is some definite improvement of performance, so that's good. There just seems to still be a few issues when actually importing/opening said projects initially. 

 




---

Click the Accept as Solution button if my answer has helped, and remember to give kudos where appropriate too!
New Here?
Join us and watch the welcome video:
Announcements
Top Kudoed Authors