users@grizzly.java.net

Upload a large file without oom with Grizzly

From: Sébastien Lorber <lorber.sebastien_at_gmail.com>
Date: Sun, 25 Aug 2013 11:35:11 +0200

Hello,

I would like to know if it's possible to upload a file with AHC / Grizzly
in streaming, I mean without loading the whole file bytes in memory.

The default behavior seems to allocate a byte[] which contans the whole
file, so it means that my server can be OOM if too many users upload a
large file in the same time.


I've tryied with a Heap and ByteBuffer memory managers, with
reallocate=true/false but no more success.

It seems the whole file content is appended wto the BufferOutputStream, and
then the underlying buffer is written.

At least this seems to be the case with AHC integration:
https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java


So, is there a way to patch AHC to stream the file so that I could
eventually consume only 20mo of heap while uploading a 500mo file?
Or is this simply impossible with Grizzly?
I didn't notice anything related to that in the documentation.

Btw in my case it is a file upload. I receive a file with CXF and have to
transmit it to a storage server (like S3). CXF doesn't consume memory
bevause it is streaming the large fle uploads to the file system, and then
provides an input stream on that file.

Thanks