Ryan thanks, it works fine, I'll make a pull request on AHC tomorrow with a
better code using the same Part classes that already exist.
I created an OutputStream that redirects to the BodyGenerator feeder.
The problem I currently have is that the feeder feeds the queue faster than
the async thread polling it :)
I need to expose a limit to that queue size or something, will work on
that, it will be better than a thread sleep to slow down the filepart
reading
2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
> Yes, something like that. I was going to tackle adding something like
> this today. I'll follow up with something you can test out.
>
>
> Sébastien Lorber wrote:
>
> Ok thanks!
>
> I think I see what I could do, probably something like that:
>
>
> FeedableBodyGenerator bodyGenerator = new FeedableBodyGenerator();
> MultipartBodyGeneratorFeeder bodyGeneratorFeeder = new
> MultipartBodyGeneratorFeeder(bodyGenerator);
> Request uploadRequest1 = new RequestBuilder("POST")
> .setUrl("url")
> .setBody(bodyGenerator)
> .build();
>
> ListenableFuture<Response> asyncRes = asyncHttpClient
> .prepareRequest(uploadRequest1)
> .execute(new AsyncCompletionHandlerBase());
>
>
> bodyGeneratorFeeder.append("param1","value1");
> bodyGeneratorFeeder.append("param2","value2");
> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
> bodyGeneratorFeeder.end();
>
> Response uploadResponse = asyncRes.get();
>
>
> Does it seem ok to you?
>
> I guess it could be interesting to provide that
> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
> people may want to achieve the same thing
>
>
>
>
>
> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>
>>
>>
>> Sébastien Lorber wrote:
>>
>>> Hello,
>>>
>>> I would like to know if it's possible to upload a file with AHC /
>>> Grizzly in streaming, I mean without loading the whole file bytes in memory.
>>>
>>> The default behavior seems to allocate a byte[] which contans the whole
>>> file, so it means that my server can be OOM if too many users upload a
>>> large file in the same time.
>>>
>>>
>>> I've tryied with a Heap and ByteBuffer memory managers, with
>>> reallocate=true/false but no more success.
>>>
>>> It seems the whole file content is appended wto the BufferOutputStream,
>>> and then the underlying buffer is written.
>>>
>>> At least this seems to be the case with AHC integration:
>>>
>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>
>>>
>>> So, is there a way to patch AHC to stream the file so that I could
>>> eventually consume only 20mo of heap while uploading a 500mo file?
>>> Or is this simply impossible with Grizzly?
>>> I didn't notice anything related to that in the documentation.
>>>
>> It's possible with the FeedableBodyGenerator. But if you're tied to
>> using Multipart uploads, you'd have to convert the multipart data to
>> Buffers manually and send using the FeedableBodyGenerator.
>> I'll take a closer look to see if this area can be improved.
>>
>>
>>> Btw in my case it is a file upload. I receive a file with CXF and have
>>> to transmit it to a storage server (like S3). CXF doesn't consume memory
>>> bevause it is streaming the large fle uploads to the file system, and then
>>> provides an input stream on that file.
>>>
>>> Thanks
>>>
>>>
>>>
>