Error Response return null while fetching more than 50,000 records over SOAPUI

[Coder]

Hi All,

i tried to fetch around 300 records and it gets working but when i tried to fetch more than 50,000 records, an error occurred "Response return null"

Please provide me some support in case someone gets stuck with it.

Thanks.

[HimanshuTayal]

@Coder : what response code you are getting, could you please check in raw response and share here?

[richie]

Hi @Coder,

It's likely youre soapui instance is running out of available resource (RAM/cpu) if you say fetching 300 records is fine, but the error response occurs when you try retrieving a result with 50000 records in it.
You might want to try optimising your laptops performance settings (setting for "best performance", stopping anything extra you dont need starting at bootup, altering you VM/swapfile settings, etc.) as well as altering soapuis memory management via the vmoptions.

Please see the following link https://www.soapui.org/getting-started/working-with-soapui/improving-memory-usage/

Ta

Rich

[Coder]

Hi richie,
I will change the memory size of the file and I will feed u back the result but are you certain that the response return null is because of the insufficient memory?

[richie]

hey @Coder,

Hahahaha...no im not certain as im not testing your system under test! but i have seen this behaviour before (when retrieved recordset is too large for my soapui instance to handle) and considering you said you had no problem retrieving records in the hundreds but failed when retrieving tens of thousands its at least a good place to start.....

For a cursory check before you mess around with the vmoptions file, why not monitor soapui's performance in windows task manager to see how much of your laptops resource is being used?

Its still worthwhile making the changes to the vmoptions file to optimise soapuis memory management based on your setup as this wont detract from performance at all if you do it right.

Ta

Rich

[Coder]

Hi richie,

the problem is still the same .however, i tried to increase the memory usage but nothing works out so please advise.

[Coder]

Hi,

anybody knows another tool instead of SOAPUI so as to retrieve more than 60,000 records?

if anybody gets stuck with it, please provide me with the recommendation regarding this case.

your quick feedback would be highly welcomed.

[richie]

Hey @Coder,

You said the things suggested didn't work, but can you confirm exactly what you did do and anything you didnt?

Did you:
1. alter the soapui vmoptions file?
2. drop all projects except relevant project from workspace?
3. alter your operating systems handling to "best performance" as well as ensuring your virtual memory is optimised?

Theres other stuff you can do to speed up/release resource on your machine (e.g
disabling services in Service Manager .msc file applet available via adminstration...you can disable all your AV etc.), but that requires a little investigation on your part.

More importantly, did you check windows taskmanager for resource usage of soapui? This would indicate if your hardware is of high enough spec to support displaying the 60000 records youre interested in. I purposefully bought an I7 cpu, 32Gigs RAM laptop a while back cos my mid range laptop wasnt cutting it for the resource intensive api/performance testing i was working on.

What spec laptop are you running?

There are a wide variety of alternative tools to do what you need, e.g. REST clients like SOATest, (and loads of different browser plugins...too many to mention although the most obvious to come to mind is Postman), theres a bunch of java libraries that allow restful testing called REST-assured (or REST-sharp built on c# libraries) and every performance test tool i can think of that ive ever used (JMeter, etc.), however depending on the spec of your laptop, switching tools wont necessarily help out. Yes i appreciate an installed app might be perceived as requiring more resource than a browser plugin for example, but 60000 records is still 60000 records, no matter what app youre using to view the records through.

I dont think the problem is soapui (although im not sitting in fromr of your laptop) i suspect its just a case that you run out of available memory when trying to read those 60k records (hence my point about monitoring windows taskmanager to view soapui resource usage).

Regarding the REST request youre using that retrieves the 60k records.....can you add REST parameters to filter the retrieved recordset? (enabling you to fetch a specific number of rows or rows up to a certain number?)

Cheers,

Rich

[Coder]

Hi richie,
Regarding the pc performance , I monitored the process and I do have 50GB RAM so there is no time out coming through but my main problem is that the error shows that the response is null as I checked the log files shown (Error request from the service gets null ) so please feed me back with other alternatives that might help overcome this issue. I can try the postman but I don't know if it might show the same issue and unable to fetch more than 60,000 records . Also, I wanna ask if this issue might come from incorrect data and the process gets blocked due to this data.
Your feedback would be quite welcomed.

[Coder]

Anybody tries to use postman previously . If so,which version is most recommended as I used version 7.3.4,7.29 and both of these versions got crashed during the processing time so please advise as I retrieve more than 60,000 records from a certain service.
Your advice would be highly appreciated.