transfer-encoding:chunked for WS request

Question

I am using soapUI 2.5.1 to test uploading documents to a WS using MTOM.&nbsp; By default I am seeing very high memory usage (10 X doc size) in the java heap both in soapUI client, and in the java server.&nbsp; My understanding is if soapUI sends the request "chunked", the server can handle the file in smaller pieces and avoid the high memory usage (and presumably soapUI can also), but I have not been able to configure my testcase in soapUI to submit this request as chunked with the transfer-encoding:chunked header.&nbsp; Is there a way to cause this to happen in soapUI?In soapUI preferences HTTP Version is 1.1, and disable chunking is not checked.&nbsp; When I execute tests against this server (including downloading documents), the response is chunked and soapUI is able to handle it.&nbsp; For the web service request that I am using to upload, I have set enable MTOM and force MTOM to true, and have added the file to upload as an attachment.&nbsp; Tried adding transfer-encoding as a HTTP property with value chunked, but since soapUI still wasn't sending the content in chunked syntax this just caused an "invalid chunk" error on the server.Thanks for any suggestions.Scott Tindal

scott_tindal · Answer

I'm not familiar with WS-Security or other features that may require access to the entire request.&nbsp; My understanding was that when MTOM was used, the file(s) included in the request were sent as MIME attachments outside of the soap envelope, I'm not sure if that helps in allowing the entire soap envelope to be accessed in memory while still streaming the attachment.&nbsp; If that is possible, then perhaps support for streaming could be available in combination with the use of MTOM.I tried the June 6 soapUI 3.0 beta 1 build, with the chunking feature enabled.&nbsp; While soapUI did use chunking to send the request, it sent the entire request as one chunk (even with a document size of 10MB), and it appears to buffer the entire request in memory, which leads to java heap problems for the soapUI JVM (similar to what I have been seeing for download tests with large documents).&nbsp; I'm not sure if this is happening because the request is sent as one chunk and so is not broken up into smaller pieces to load into memory, or if it just a case where the entire request is always loaded into memory by soapUI.In either case (chunking or not), a key requirement for me is to have soapUI not buffer the entire request in memory.&nbsp; I need to test uploading documents up to a size of 1GB, currently java heap usage is roughly 10x the document size so the largest document I can upload is about 100MB (with just one client thread).Thanks,Scott

smartbear_suppo · Answer

Hello,The implementation of HttpClient that soapUI uses controls whether to use chunking or not. SoapUI lets you disable chunking, but unfortunately not to force it. I am not sure what triggers HttpClient to use Chunking. Perhaps the option to force chunking will be included in a future version of soapUI.Regards,Daineviware support

scott_tindal · Answer

Is it possible to use a groovy script to make this type of call on the HttpURLConnection object that will be used by a subsequent request step?&nbsp; The alternative workaround that I will try is to implement the WS call entirely within a groovy script step, but I'd rather avoid this if possible as it eliminates the main benefit of using soapUI to simplify WS testing.

smartbear_suppo · Answer

Hello,I've looked in to this a bit more, and have found a way to force chunking. An idea now is to remove the checkbox to disable chunking in the HTTP Preferences, and replace it with an integer value where you can specify a threshold request size. Any request larger than the value will be chunked and any request smaller will not. Setting this to 0 would cause all requests to use chunking and setting it to -1 would completely disable chunking. What do you think of this approach, would it suit your needs well? If you have any other ideas that you think would be better, please let us know.Regards,Daineviware support

scott_tindal · Answer

Hi Dain,Sorry for the delay in responding, I've been sidetracked for a few weeks.I think this would be a useful addition to soapUI, it would address my original requirement to execute the test with chunking enabled.&nbsp; Perhaps an incremental improvement would be to make this a property of the TestRequest, rather than a global preference, so that chunking could be enabled/disabled, and the size specified, on a per request basis.I have an additional related question.&nbsp; As an alternative to enabling chunking, it should also be possible to avoid the memory issues I was experiencing by streaming the upload.&nbsp; I'm told that for a java client, that is sending requests directly using the HttpUrlConnection class, calling setFixedLengthStreamingMode (ContentLength) will enable streaming.&nbsp; For a java client using the jaxws library, it's not possible to call the above method directly on the underlying HttpUrlConnection, but a jaxws chunksize property can be set to cause it to call setChunkedStreamingMode(chunkLength) on the underlying HttpUrlConnection.I was wondering if you could describe how soapUI is sending the WS requests internally, and whether it would be possible to also expose a configuration property that would enable streaming for the request?&nbsp; Thanks,Scott

Forum Discussion

Recent Discussions

Shared workspace

No Fluff Just Real Stuff - latest newsletter from the DevRel Team at SmartBear

Which current version of POI bin files is compatible with SOAP UI 5.5 free version ?

Related Content

Request Body Missing in Raw Request view

[LoadUIWeb] Does LoadUIWeb support Transfer-Encoding:chunked HTTP header?

POST request return 400 Bad request