Forum Discussion
richie
Community Hero
Hey Coder,
You said the things suggested didn't work, but can you confirm exactly what you did do and anything you didnt?
Did you:
1. alter the soapui vmoptions file?
2. drop all projects except relevant project from workspace?
3. alter your operating systems handling to "best performance" as well as ensuring your virtual memory is optimised?
Theres other stuff you can do to speed up/release resource on your machine (e.g
disabling services in Service Manager .msc file applet available via adminstration...you can disable all your AV etc.), but that requires a little investigation on your part.
More importantly, did you check windows taskmanager for resource usage of soapui? This would indicate if your hardware is of high enough spec to support displaying the 60000 records youre interested in. I purposefully bought an I7 cpu, 32Gigs RAM laptop a while back cos my mid range laptop wasnt cutting it for the resource intensive api/performance testing i was working on.
What spec laptop are you running?
There are a wide variety of alternative tools to do what you need, e.g. REST clients like SOATest, (and loads of different browser plugins...too many to mention although the most obvious to come to mind is Postman), theres a bunch of java libraries that allow restful testing called REST-assured (or REST-sharp built on c# libraries) and every performance test tool i can think of that ive ever used (JMeter, etc.), however depending on the spec of your laptop, switching tools wont necessarily help out. Yes i appreciate an installed app might be perceived as requiring more resource than a browser plugin for example, but 60000 records is still 60000 records, no matter what app youre using to view the records through.
I dont think the problem is soapui (although im not sitting in fromr of your laptop) i suspect its just a case that you run out of available memory when trying to read those 60k records (hence my point about monitoring windows taskmanager to view soapui resource usage).
Regarding the REST request youre using that retrieves the 60k records.....can you add REST parameters to filter the retrieved recordset? (enabling you to fetch a specific number of rows or rows up to a certain number?)
Cheers,
Rich
You said the things suggested didn't work, but can you confirm exactly what you did do and anything you didnt?
Did you:
1. alter the soapui vmoptions file?
2. drop all projects except relevant project from workspace?
3. alter your operating systems handling to "best performance" as well as ensuring your virtual memory is optimised?
Theres other stuff you can do to speed up/release resource on your machine (e.g
disabling services in Service Manager .msc file applet available via adminstration...you can disable all your AV etc.), but that requires a little investigation on your part.
More importantly, did you check windows taskmanager for resource usage of soapui? This would indicate if your hardware is of high enough spec to support displaying the 60000 records youre interested in. I purposefully bought an I7 cpu, 32Gigs RAM laptop a while back cos my mid range laptop wasnt cutting it for the resource intensive api/performance testing i was working on.
What spec laptop are you running?
There are a wide variety of alternative tools to do what you need, e.g. REST clients like SOATest, (and loads of different browser plugins...too many to mention although the most obvious to come to mind is Postman), theres a bunch of java libraries that allow restful testing called REST-assured (or REST-sharp built on c# libraries) and every performance test tool i can think of that ive ever used (JMeter, etc.), however depending on the spec of your laptop, switching tools wont necessarily help out. Yes i appreciate an installed app might be perceived as requiring more resource than a browser plugin for example, but 60000 records is still 60000 records, no matter what app youre using to view the records through.
I dont think the problem is soapui (although im not sitting in fromr of your laptop) i suspect its just a case that you run out of available memory when trying to read those 60k records (hence my point about monitoring windows taskmanager to view soapui resource usage).
Regarding the REST request youre using that retrieves the 60k records.....can you add REST parameters to filter the retrieved recordset? (enabling you to fetch a specific number of rows or rows up to a certain number?)
Cheers,
Rich
Coder
4 years agoOccasional Contributor
Hi richie,
Regarding the pc performance , I monitored the process and I do have 50GB RAM so there is no time out coming through but my main problem is that the error shows that the response is null as I checked the log files shown (Error request from the service gets null ) so please feed me back with other alternatives that might help overcome this issue. I can try the postman but I don't know if it might show the same issue and unable to fetch more than 60,000 records . Also, I wanna ask if this issue might come from incorrect data and the process gets blocked due to this data.
Your feedback would be quite welcomed.
Regarding the pc performance , I monitored the process and I do have 50GB RAM so there is no time out coming through but my main problem is that the error shows that the response is null as I checked the log files shown (Error request from the service gets null ) so please feed me back with other alternatives that might help overcome this issue. I can try the postman but I don't know if it might show the same issue and unable to fetch more than 60,000 records . Also, I wanna ask if this issue might come from incorrect data and the process gets blocked due to this data.
Your feedback would be quite welcomed.
- Coder4 years agoOccasional ContributorAnybody tries to use postman previously . If so,which version is most recommended as I used version 7.3.4,7.29 and both of these versions got crashed during the processing time so please advise as I retrieve more than 60,000 records from a certain service.
Your advice would be highly appreciated. - Coder4 years agoOccasional ContributorHi all,
I found that the postman tool is not a recommended solution to fetch more than 65,000 records but it might be useful for small data. As I tried to fetch large data, postman kept crashed suddenly and didn't complete the process . However, it fetches small data (200-1000 records) but it didn't exceed more than that.
Anybody gets stuck with it and have a new approach for this case, please post it to be accepted as a new alternative approach. - HimanshuTayal4 years agoCommunity Hero
Coder : The URL which you are hitting is private or public?
If it is public could you please provide it to me so that i can try at my end and will let you know if i found something helpful?
- richie4 years agoCommunity HeroHey Coder,
You say your request returns "null", but can you explain what that actually means?
When you check taskmanager, does the vm resource usage just continue to increase without plateauing?
Is there a way to filter the retrieved recordset so you only fetch a certain number of rows? The reason im asking this is that an exposed api that allows the user to retrieve 60k records is not really helpful to the user, so id expect perhaps a query parameter enabling you to filter the query to be avialable....or at least this is what ive found before now.
When you submit the request do you get any sort of response (http response code) or does it just hang and never finish?
Most testing tools are gonna struggle to retrieve 60k records cos unless theyve been opyimised to handle this number of records, the whole 60k recordset needs to be loaded into memory to enable it to display it.
If youre not getting a response at all and the request for the 60k records doesnt complete, id be interested in viewing the logging of your http/application server cos the problem could actually be at that end and NOT at the ReadyAPI!/SoapUI end. From your current description im starting to suspect the issue is the app server.
Ta
Rich