Forum Discussion

Cata's avatar
Cata
Occasional Contributor
4 years ago

Parsing large JSON responses

Hi there,

I have some API requests that return a very large JSON response (about 8mb/ 400k lines) and I'm having trouble parsing them.
So I'm using a Groovy script to try and parse the response but when I do this the ReadyApi tool freezes completely and there's nothing else I can do other than just shut it down.

I've looked around here and I found this topic that mentioned something about increasing the resources ReadyAPI is using and I've tried that. Apparently my heap size was already at Xmx7992m, by default (my machine has a total of 16gb of RAM). Not sure if that's the issue.

I've also tried to use the Gson java library that supposedly helps with the parsing of such large files but still no use. ReadyAPI freezes and after a few minutes of inactivity I receive an error that's not even telling me anything. Again at that point the app just freezes and I can't close the error window or check the tool's error logs.


I'm sure there are others who have run into this type of issue. How did you manage to resolve them?


Thanks

Function Tests
REST
Scripting
  • richie's avatar
    richie
    4 years ago

    Hey Cata,

     

     

    so your Xmx is 7992Megs - so that should be enough to cover off 6 Gigs worth of memory usage.  Take it up a little further perhaps?

     

    Have you checked the logging at all?  I think you said the whole app just hangs and crashes - but perhaps content was still being written to the logs.

     

    Im on my work Mac at the moment so cant check my Windows laptop (where my ReadyAPI instance is) - but I think (going on memory) the logs write to files in something like C:\Users\<profile>\ and there's a I think either a hidden ReadyAPI folder that contains the log files.

     

    Helpful detail may have been written to the logs - but for now I'm still going with my suggestion that youre running out of resource.

     

    You could always execute the tests in headless mode?  i.e. execute the test via testRunner (command line) - that will minimise the RAM usage quite a bit.

     

    ta

     

    Rich

  • richie's avatar
    richie
    Community Hero
    4 years ago
    Hey Cata,

    What version of ReadyAPI are you using?

    About a year ago i was testing a DB migration that returned 10000 records and i had no problems.

    Although i suppose, it also depends on what else is memory resident when youre executing your test.

    Have you tried monitoring task manager when you try making the call? That'll indicate how much resource is being used and provide evidence if your machine is of high enough spec to successfully make the call.

    Cheers,

    Rich
    • Cata's avatar
      Cata
      Occasional Contributor
      4 years ago

      Hi richie,

      Thanks for the reply.
      I'm currently running the latest version of ReadyAPI, 3.10.1. 
      When I run the groovy script that's supposed to parse the JSON response, this is the resource consumption that task managers shows:


      As soon as I start running the test, the RAM jumps to about 4gb and then it kind of stops around that value, 6gb. CPU consumption alternates between 10-20%. Funny thing is, after that empty error window appears the resource consumption doesn't die down, it still remains at those max values.

      Should I try to increase the heap size? Not sure exactly what values I should use though. My machine has a max of 16gb RAM installed.

      • richie's avatar
        richie
        Community Hero
        4 years ago

        Hey Cata,

         

         

        so your Xmx is 7992Megs - so that should be enough to cover off 6 Gigs worth of memory usage.  Take it up a little further perhaps?

         

        Have you checked the logging at all?  I think you said the whole app just hangs and crashes - but perhaps content was still being written to the logs.

         

        Im on my work Mac at the moment so cant check my Windows laptop (where my ReadyAPI instance is) - but I think (going on memory) the logs write to files in something like C:\Users\<profile>\ and there's a I think either a hidden ReadyAPI folder that contains the log files.

         

        Helpful detail may have been written to the logs - but for now I'm still going with my suggestion that youre running out of resource.

         

        You could always execute the tests in headless mode?  i.e. execute the test via testRunner (command line) - that will minimise the RAM usage quite a bit.

         

        ta

         

        Rich