Excellent! Looking forward to the pull request!
Sébastien Lorber wrote:
> Ryan thanks, it works fine, I'll make a pull request on AHC tomorrow
> with a better code using the same Part classes that already exist.
>
> I created an OutputStream that redirects to the BodyGenerator feeder.
>
> The problem I currently have is that the feeder feeds the queue faster
> than the async thread polling it :)
> I need to expose a limit to that queue size or something, will work on
> that, it will be better than a thread sleep to slow down the filepart
> reading
>
>
> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com
> <mailto:ryan.lubke_at_oracle.com>>
>
> Yes, something like that. I was going to tackle adding something
> like this today. I'll follow up with something you can test out.
>
>
> Sébastien Lorber wrote:
>> Ok thanks!
>>
>> I think I see what I could do, probably something like that:
>>
>>
>> FeedableBodyGenerator bodyGenerator = new
>> FeedableBodyGenerator();
>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder = new
>> MultipartBodyGeneratorFeeder(bodyGenerator);
>> Request uploadRequest1 = new RequestBuilder("POST")
>> .setUrl("url")
>> .setBody(bodyGenerator)
>> .build();
>>
>> ListenableFuture<Response> asyncRes = asyncHttpClient
>> .prepareRequest(uploadRequest1)
>> .execute(new AsyncCompletionHandlerBase());
>>
>>
>> bodyGeneratorFeeder.append("param1","value1");
>> bodyGeneratorFeeder.append("param2","value2");
>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>> bodyGeneratorFeeder.end();
>>
>> Response uploadResponse = asyncRes.get();
>>
>>
>> Does it seem ok to you?
>>
>> I guess it could be interesting to provide that
>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some
>> other people may want to achieve the same thing
>>
>>
>>
>>
>>
>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com
>> <mailto:ryan.lubke_at_oracle.com>>
>>
>>
>>
>> Sébastien Lorber wrote:
>>
>> Hello,
>>
>> I would like to know if it's possible to upload a file
>> with AHC / Grizzly in streaming, I mean without loading
>> the whole file bytes in memory.
>>
>> The default behavior seems to allocate a byte[] which
>> contans the whole file, so it means that my server can be
>> OOM if too many users upload a large file in the same time.
>>
>>
>> I've tryied with a Heap and ByteBuffer memory managers,
>> with reallocate=true/false but no more success.
>>
>> It seems the whole file content is appended wto the
>> BufferOutputStream, and then the underlying buffer is
>> written.
>>
>> At least this seems to be the case with AHC integration:
>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>
>>
>> So, is there a way to patch AHC to stream the file so
>> that I could eventually consume only 20mo of heap while
>> uploading a 500mo file?
>> Or is this simply impossible with Grizzly?
>> I didn't notice anything related to that in the
>> documentation.
>>
>> It's possible with the FeedableBodyGenerator. But if you're
>> tied to using Multipart uploads, you'd have to convert the
>> multipart data to Buffers manually and send using the
>> FeedableBodyGenerator.
>> I'll take a closer look to see if this area can be improved.
>>
>>
>> Btw in my case it is a file upload. I receive a file with
>> CXF and have to transmit it to a storage server (like
>> S3). CXF doesn't consume memory bevause it is streaming
>> the large fle uploads to the file system, and then
>> provides an input stream on that file.
>>
>> Thanks
>>
>>
>>
>