Forum Discussion

Scott_Tindal's avatar
Scott_Tindal
Occasional Contributor
15 years ago

Re: Avoiding high memory use when downloading large documents

I'm using soapUI Pro 2.5.1, executing command line loadtestrunner.bat to run a load test that will repeatedly download large (100MB) documents using a web
service call.  
- I've modified loadtestrunner.bat to have -Xms1024m -Xmx1024m so that the java heap is increased to 1GB
- The web service supports MTOM, and I have configured the request properties to have Enable MTOM and Force MTOM to true.  
- In the testcase options, I have also set 'discards successful testresults to preserve memory'.  

It appears that soapUI retrieves the entire document into memory during a download, with the above setting previous downloads can be garbage collected so I don't get OutOfMemory errors, but when running a number of threads, soapUI still uses up java heap memory quite quickly resulting in a lot of time spent in garbage collection (during a 5 thread test downloading 10MB documents, JVM throughput was about 66% (i.e. 34% of time spent in GC)).  Is there a way to have soapUI stream this type of download to a temp file rather than storing it all in memory, so that this type of test could be executed with less GC overhead?  This has a big effect on the download time.

I tried setting the "Dump File" TestRequest property to a file, but this didn't affect the memory usage, and only one file can be specified so I'm not sure how this property would help with multi threaded test.

Thanks,
Scott Tindal

6 Replies

  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    Hi!

    could you try setting the dump-file to something like data-${context.ThreadIndex}.dat and the Max Size property to 0, this should load no data into memory and all to the file (you might need the latest nightly build for this; http://www.eviware.com/nightly-builds)

    Let me know!

    regards,

    /Ole
    eviware.com
  • Scott_Tindal's avatar
    Scott_Tindal
    Occasional Contributor
    Hi,

    I downloaded the June 3 2.5.2 build, and set the dump file parameter as you described, but I'm still seeing the high memory usage.  With a single thread, downloading a 100MB file, java tenured generation memory usage jumps up over 500MB.  I can send a .hprof memory dump taken with the Sun jvisualvm tool if that would help.

    Also, the dump file parameter didn't properly parse that variable, I set it to:
    c:/public/DLTempFiles/data-${context.ThreadIndex}.dat
    But the file created was:
    C:\public\DLTempFiles\data-.dat

    I also downloaded the June 3 3.0 beta 1 build (the win32 standalone bin zip: http://www.eviware.com/nightly-builds/2 ... ne-bin.zip), but when I try to execute soapui-pro.bat it I get this error:
    Exception in thread "XpcEventThread" javax.management.RuntimeErrorException
            at com.teamdev.xpcom.Xpcom.a(SourceFile:40)
            at com.teamdev.xpcom.a.run(SourceFile:119)
            at com.teamdev.xpcom.d.run(SourceFile:436)
            at com.teamdev.xpcom.impl.awt.d$a.run(SourceFile:68)
            at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.Error: java.io.FileNotFoundException: GRE not found
            ... 5 more
    Caused by: java.io.FileNotFoundException: GRE not found
            at org.mozilla.xpcom.Mozilla.getGREPathWithProperties(Mozilla.java:177)
            ... 5 more

    Googling this error suggests I need a Software\\mozilla.org\\GRE registry key, can I create this, or do I need to run the full installer for soapUI 3.0?

    Thanks,
    Scott
  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    Hi Scott,

    ok.. i'll profile into this as soon as possible..

    the property expansion should just have been ${ThreadIndex}, sorry about that.

    Regarding the standalone distro, we'll check into it.. If you can use the installer that would probably work better, otherwise just set a GRE_HOME environment property pointing to the xulrunner directory in your soapUI installation.

    regards!

    /Ole
    eviware.com
  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    Hi,

    Regarding the standalone distribution,please download it from upcoming nightly build and that should solve the problem .
    Notice that soapui.bat file contains commented line where soapui.jxbrowser.disable property is set to true. If you have any further problems with the browser (i.e. GRE_HOME) try uncommenting that line and starting again.

    regards,

    /Dragica
    eviware.com
  • Scott_Tindal's avatar
    Scott_Tindal
    Occasional Contributor
    Hi,

    I downloaded the June 6 soapUI Pro 3.0 beta 1, and was able to run it and did not get the GRE error.

    Using ${ThreadIndex} as the variable in the file name is working to generate a unique file name per client thread.

    However, running my download testcase in this 3.0 build, with the dump-file parameter set in this way, I am still seeing the high java heap memory usage in soapUI JVM.

    Scott
  • Scott_Tindal's avatar
    Scott_Tindal
    Occasional Contributor
    Hi,

    Is there any update about the high memory usage when uploading or downloading large documents in soapui?

    Thanks,
    Scott