transfer-encoding:chunked for WS request

I'm not familiar with WS-Security or other features that may require access to the entire request.  My understanding was that when MTOM was used, the file(s) included in the request were sent as MIME attachments outside of the soap envelope, I'm not sure if that helps in allowing the entire soap envelope to be accessed in memory while still streaming the attachment.  If that is possible, then perhaps support for streaming could be available in combination with the use of MTOM.

I tried the June 6 soapUI 3.0 beta 1 build, with the chunking feature enabled.  While soapUI did use chunking to send the request, it sent the entire request as one chunk (even with a document size of 10MB), and it appears to buffer the entire request in memory, which leads to java heap problems for the soapUI JVM (similar to what I have been seeing for download tests with large documents).  I'm not sure if this is happening because the request is sent as one chunk and so is not broken up into smaller pieces to load into memory, or if it just a case where the entire request is always loaded into memory by soapUI.

In either case (chunking or not), a key requirement for me is to have soapUI not buffer the entire request in memory.  I need to test uploading documents up to a size of 1GB, currently java heap usage is roughly 10x the document size so the largest document I can upload is about 100MB (with just one client thread).

Thanks,
Scott

Hello,

The implementation of HttpClient that soapUI uses controls whether to use chunking or not. SoapUI lets you disable chunking, but unfortunately not to force it. I am not sure what triggers HttpClient to use Chunking. Perhaps the option to force chunking will be included in a future version of soapUI.

Regards,
Dain
eviware support

Is it possible to use a groovy script to make this type of call on the HttpURLConnection object that will be used by a subsequent request step?  The alternative workaround that I will try is to implement the WS call entirely within a groovy script step, but I'd rather avoid this if possible as it eliminates the main benefit of using soapUI to simplify WS testing.

Hello,

I've looked in to this a bit more, and have found a way to force chunking. An idea now is to remove the checkbox to disable chunking in the HTTP Preferences, and replace it with an integer value where you can specify a threshold request size. Any request larger than the value will be chunked and any request smaller will not. Setting this to 0 would cause all requests to use chunking and setting it to -1 would completely disable chunking. What do you think of this approach, would it suit your needs well? If you have any other ideas that you think would be better, please let us know.

Regards,
Dain
eviware support

Hi Dain,

Sorry for the delay in responding, I've been sidetracked for a few weeks.

I think this would be a useful addition to soapUI, it would address my original requirement to execute the test with chunking enabled.  Perhaps an incremental improvement would be to make this a property of the TestRequest, rather than a global preference, so that chunking could be enabled/disabled, and the size specified, on a per request basis.

I have an additional related question.  As an alternative to enabling chunking, it should also be possible to avoid the memory issues I was experiencing by streaming the upload.  I'm told that for a java client, that is sending requests directly using the HttpUrlConnection class, calling setFixedLengthStreamingMode (ContentLength) will enable streaming.  For a java client using the jaxws library, it's not possible to call the above method directly on the underlying HttpUrlConnection, but a jaxws chunksize property can be set to cause it to call setChunkedStreamingMode(chunkLength) on the underlying HttpUrlConnection.

I was wondering if you could describe how soapUI is sending the WS requests internally, and whether it would be possible to also expose a configuration property that would enable streaming for the request? 

Thanks,
Scott

Hi Scott,

thanks for all your thoughts and insights!

soapUI 3.0 improved on the global chunking setting; it allows you to set a chunking limit; if the size of the request is larger, it will be chunked. Moving this to the request level as well would be relatively easy if you find it motivated.

soapUI uses the HttpClient library internally for sending requests, which supports streaming (via the requestentity related classes I think), but this would require some work since we still need to support property-expansions, ws-security, etc.. which require access to parts or the entire request (for example for signing..)

looking forward to your ideas!

/Ole
eviware.com