Forum Discussion

Cata's avatar
Cata
Occasional Contributor
4 years ago

Parsing large JSON responses

Hi there, I have some API requests that return a very large JSON response (about 8mb/ 400k lines) and I'm having trouble parsing them. So I'm using a Groovy script to try and parse the response b...
Function Tests
REST
Scripting
  • richie's avatar
    richie
    4 years ago

    Hey Cata,

     

     

    so your Xmx is 7992Megs - so that should be enough to cover off 6 Gigs worth of memory usage.  Take it up a little further perhaps?

     

    Have you checked the logging at all?  I think you said the whole app just hangs and crashes - but perhaps content was still being written to the logs.

     

    Im on my work Mac at the moment so cant check my Windows laptop (where my ReadyAPI instance is) - but I think (going on memory) the logs write to files in something like C:\Users\<profile>\ and there's a I think either a hidden ReadyAPI folder that contains the log files.

     

    Helpful detail may have been written to the logs - but for now I'm still going with my suggestion that youre running out of resource.

     

    You could always execute the tests in headless mode?  i.e. execute the test via testRunner (command line) - that will minimise the RAM usage quite a bit.

     

    ta

     

    Rich

richie's avatar
richie
Community Hero
4 years ago
Hey Cata,

What version of ReadyAPI are you using?

About a year ago i was testing a DB migration that returned 10000 records and i had no problems.

Although i suppose, it also depends on what else is memory resident when youre executing your test.

Have you tried monitoring task manager when you try making the call? That'll indicate how much resource is being used and provide evidence if your machine is of high enough spec to successfully make the call.

Cheers,

Rich
  • Cata's avatar
    Cata
    Occasional Contributor
    4 years ago

    Hi richie,

    Thanks for the reply.
    I'm currently running the latest version of ReadyAPI, 3.10.1. 
    When I run the groovy script that's supposed to parse the JSON response, this is the resource consumption that task managers shows:


    As soon as I start running the test, the RAM jumps to about 4gb and then it kind of stops around that value, 6gb. CPU consumption alternates between 10-20%. Funny thing is, after that empty error window appears the resource consumption doesn't die down, it still remains at those max values.

    Should I try to increase the heap size? Not sure exactly what values I should use though. My machine has a max of 16gb RAM installed.

    • richie's avatar
      richie
      Community Hero
      4 years ago

      Hey Cata,

       

       

      so your Xmx is 7992Megs - so that should be enough to cover off 6 Gigs worth of memory usage.  Take it up a little further perhaps?

       

      Have you checked the logging at all?  I think you said the whole app just hangs and crashes - but perhaps content was still being written to the logs.

       

      Im on my work Mac at the moment so cant check my Windows laptop (where my ReadyAPI instance is) - but I think (going on memory) the logs write to files in something like C:\Users\<profile>\ and there's a I think either a hidden ReadyAPI folder that contains the log files.

       

      Helpful detail may have been written to the logs - but for now I'm still going with my suggestion that youre running out of resource.

       

      You could always execute the tests in headless mode?  i.e. execute the test via testRunner (command line) - that will minimise the RAM usage quite a bit.

       

      ta

       

      Rich

      • Cata's avatar
        Cata
        Occasional Contributor
        4 years ago

        Hey richie 

        I got it to work in the end. It was my fault. I made a mistake in the groovy code and instead of saying 'parseText' I had written simply 'parse' and that's the reason why it got stuck. I corrected it and the call now seems to run in a split second, no delay or anything.
        Thanks a lot for the help though. Much appreciated.