Parsing large JSON responses

[Cata]

Hi there,

I have some API requests that return a very large JSON response (about 8mb/ 400k lines) and I'm having trouble parsing them.
So I'm using a Groovy script to try and parse the response but when I do this the ReadyApi tool freezes completely and there's nothing else I can do other than just shut it down.

I've looked around here and I found this topic that mentioned something about increasing the resources ReadyAPI is using and I've tried that. Apparently my heap size was already at Xmx7992m, by default (my machine has a total of 16gb of RAM). Not sure if that's the issue.

I've also tried to use the Gson java library that supposedly helps with the parsing of such large files but still no use. ReadyAPI freezes and after a few minutes of inactivity I receive an error that's not even telling me anything. Again at that point the app just freezes and I can't close the error window or check the tool's error logs.


I'm sure there are others who have run into this type of issue. How did you manage to resolve them?


Thanks

[richie]

Hey Cata,

What version of ReadyAPI are you using?

About a year ago i was testing a DB migration that returned 10000 records and i had no problems.

Although i suppose, it also depends on what else is memory resident when youre executing your test.

Have you tried monitoring task manager when you try making the call? That'll indicate how much resource is being used and provide evidence if your machine is of high enough spec to successfully make the call.

Cheers,

Rich

[Cata]

Hi richie,

Thanks for the reply.
I'm currently running the latest version of ReadyAPI, 3.10.1. 
When I run the groovy script that's supposed to parse the JSON response, this is the resource consumption that task managers shows:


As soon as I start running the test, the RAM jumps to about 4gb and then it kind of stops around that value, 6gb. CPU consumption alternates between 10-20%. Funny thing is, after that empty error window appears the resource consumption doesn't die down, it still remains at those max values.

Should I try to increase the heap size? Not sure exactly what values I should use though. My machine has a max of 16gb RAM installed.

[richie · Accepted solution]

Hey Cata,

so your Xmx is 7992Megs - so that should be enough to cover off 6 Gigs worth of memory usage.  Take it up a little further perhaps?

Have you checked the logging at all?  I think you said the whole app just hangs and crashes - but perhaps content was still being written to the logs.

Im on my work Mac at the moment so cant check my Windows laptop (where my ReadyAPI instance is) - but I think (going on memory) the logs write to files in something like C:\Users\<profile>\ and there's a I think either a hidden ReadyAPI folder that contains the log files.

Helpful detail may have been written to the logs - but for now I'm still going with my suggestion that youre running out of resource.

You could always execute the tests in headless mode?  i.e. execute the test via testRunner (command line) - that will minimise the RAM usage quite a bit.

ta

Rich

[Cata]

Hey @richie 

I got it to work in the end. It was my fault. I made a mistake in the groovy code and instead of saying 'parseText' I had written simply 'parse' and that's the reason why it got stuck. I corrected it and the call now seems to run in a split second, no delay or anything.
Thanks a lot for the help though. Much appreciated.