Yes I think it would, so that I can feed the queue at once
One thread will be locked during the feeding for nothing but it's not a
real problem in my usecase.
2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
> Would having a different listener that will be notified once async
> transferring has been started work better for you?
>
> Something like:
>
> onAsyncTransferInitiated() {
> // invoke your feed method
> }
>
> ?
>
>
>
> Sébastien Lorber wrote:
>
> Unfortunatly I won't be able to use the Feeder non-blocking stuff for now,
> because of how the multipart request in handled in AHC
>
>
> Here's my feeding method:
>
> public void feed() throws IOException {
> Part[] partsArray = parts.toArray(new Part[parts.size()]);
> try ( OutputStream outputStream = createFeedingOutputStream() ) {
> Part.sendParts(outputStream,partsArray,multipartBoundary);
> } catch (Exception e) {
> throw new IllegalStateException("Unable to feed the
> FeedableBodyGenerator",e);
> }
> }
>
>
> As you can see, the multipart Parts array can only be pushed to the
> OutputStream, I don't have any way to "pull" the data when the canFeed()
> method is triggered.
>
>
> But I've seen that there's a com.ning.http.multipart.MultipartBody#read
> that seems to provide a memory efficient way to pull data from a Multipart
> body...
>
> Should see what I come up with this
>
>
> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>
>> It seems the Feeder is highly recommended but not mandatory so I tried
>> without.
>>
>> With my existing code it seems there is a synchronization problem.
>>
>>
>> The feeding threads get locked to the prematureFeed.get();
>>
>> So the Grizzly kernel threads are unable to acquire the lock required to
>> enter the initializeAsynchronousTransfer method
>>
>>
>>
>> Will try with an implementation of Feeder
>>
>>
>>
>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>
>>> Hmmm it seems I have a problem with one of your maven plugins. I'll try
>>> to bypass it, but for info:
>>>
>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>> [WARNING]
>>> [WARNING] Some problems were encountered while building the effective
>>> settings
>>> [WARNING] 'profiles.profile[default].repositories.repository.id' must
>>> be unique but found duplicate repository with id fullsix-maven-repository @
>>> /home/slorber/.m2/settings.xml
>>> [WARNING]
>>> [INFO] Scanning for projects...
>>> [INFO]
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO]
>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>> async-http-client ---
>>> [INFO]
>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven) @
>>> async-http-client ---
>>> [INFO]
>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-versions) @
>>> async-http-client ---
>>> [INFO]
>>> [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
>>> async-http-client ---
>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>> [INFO] skip non existing resourceDirectory
>>> /home/slorber/Bureau/ahc2/src/main/resources
>>> [INFO]
>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
>>> async-http-client ---
>>> [INFO] Compiling 158 source files to
>>> /home/slorber/Bureau/ahc2/target/classes
>>> [INFO]
>>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>>> (check-java-1.5-compat) @ async-http-client ---*
>>> *[INFO] Checking unresolved references to
>>> org.codehaus.mojo.signature:java15:1.0*
>>> *[ERROR] Undefined reference:
>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
>>> *
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 8.747s
>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>> [INFO] Final Memory: 30M/453M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> *[ERROR] Failed to execute goal
>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]*
>>> [ERROR]
>>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>>> -e switch.
>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>> [ERROR]
>>> [ERROR] For more information about the errors and possible solutions,
>>> please read the following articles:
>>> [ERROR] [Help 1]
>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>
>>>
>>>
>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>
>>>> Ok thank you, I'll try to implement that today and will give you my
>>>> feedback :)
>>>>
>>>>
>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>
>>>>> Okay,
>>>>>
>>>>> I've committed my initial changes to the AHC repository. Here's a
>>>>> summary of the changes:
>>>>>
>>>>>
>>>>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>> - Don't allow queueing of data before initiateAsyncTransfer has been
>>>>> invoked. In low memory
>>>>> heaps, this could lead to an OOM if the source is feeding too fast.
>>>>> The new behavior is to
>>>>> block until initiateAsyncTransfer is called, at which time the blocked
>>>>> thread may proceed with
>>>>> the feed operation.
>>>>> - Introduce the concept of a Feeder. Implementations are responsible,
>>>>> at a high level, for:
>>>>> + letting the provider know that data is available to be fed without
>>>>> blocking
>>>>> + allowing the registration of a callback that the Feeder
>>>>> implementation may invoke
>>>>> to signal that more data is available, if it wasn't available at a
>>>>> previous point in time.
>>>>> - When using a Feeder with a secure request, the SSL handshake will be
>>>>> kicked off by the
>>>>> initiateAsyncTransfer call, but feeding of data will not occur until
>>>>> the handshake is complete.
>>>>> This is necessary as the SSLFilter will queue up all writes until the
>>>>> handshake is complete,
>>>>> and currently, the buffer isn't tied in with the transport flow
>>>>> control mechanism.
>>>>> NOTE: This new SSL behavior is not currently applied when invoking the
>>>>> feed() method
>>>>> outside the context of a Feeder. Still need to address that.
>>>>> - Exposed configuration of the async write queue limit through the
>>>>> FeedableBodyGenerator.
>>>>> This is an improvement on using a TransportCustomizer as any
>>>>> configuration there is
>>>>> transport-wide, and therefor applied to all Connections. By exposing
>>>>> it here, each feeder
>>>>> may have a different byte limit.
>>>>> - Improved documentation for this class*
>>>>>
>>>>> I recommend reading through the javadoc comments in the source [1] for
>>>>> FeedableBodyGenerator (comments welcome).
>>>>> Additionally, I would re-work your code to leverage the Feeder instead
>>>>> of calling feed() directly.
>>>>>
>>>>> If you have issues implementing Feeder, do let us know.
>>>>>
>>>>> If you have additional questions, again, let us know.
>>>>>
>>>>> Thanks,
>>>>> -rl
>>>>>
>>>>> [1]
>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>
>>>>>
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>
>>>>> So in the end I've end up with an implementation that's working for me.
>>>>>
>>>>>
>>>>> I think there are 2 bugs:
>>>>>
>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue if the
>>>>> initialize(ctx) method is not called fast enough.
>>>>> This can be solved by using a BlockingQueue of size 1 and the put()
>>>>> method.
>>>>>
>>>>> 2) Once the context is injected, the FeedableBodyGenerator flushes the
>>>>> queue.
>>>>> The matter is that if the connection is new, not warmed up by a
>>>>> previous request, then the SSL handshake is not done yet, and it seems that
>>>>> the bytes are accumulated in some part of the SSL filter which doesn't
>>>>> deliver them to the connection until the handshake has completed,
>>>>> so c.canWrite() continues to return true.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I have replaced some part of the FeedableBodyGenerator to test this
>>>>> and it works pretty fine. See what I have changed:
>>>>>
>>>>> 1)
>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>
>>>>>
>>>>> 2)
>>>>> public void feed(final Buffer buffer, final boolean last) throws
>>>>> IOException {
>>>>> try {
>>>>> queue.put(new BodyPart(buffer, last));
>>>>> } catch (InterruptedException e) {
>>>>> throw new RuntimeException(e);
>>>>> }
>>>>> queueSize.incrementAndGet();
>>>>>
>>>>> if (context != null) {
>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>> flushQueue(true);
>>>>> }
>>>>> }
>>>>>
>>>>> private void blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>>> fcc) {
>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>> try { Thread.sleep(10); } catch ( Exception e ) { throw new
>>>>> RuntimeException(e); }
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> private boolean connectionIsReadyToWrite(FilterChainContext fcc) {
>>>>> Connection connection = fcc.getConnection();
>>>>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>>>>> return sslEngine != null && !SSLUtils.isHandshaking(sslEngine);
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> What do you think?
>>>>>
>>>>>
>>>>> We had come to similar conclusions on this end. I'm still working
>>>>> through testing the idea I mentioned previously (took longer than I
>>>>> expected - sorry).
>>>>> I hope to have something for you to test very soon.
>>>>>
>>>>> Note that it will be taking the above into account as well.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>
>>>>>>
>>>>>> I have tried to put a while ( context == null ) Thread.sleep but it
>>>>>> doesn't seem to work, when the context gets injected, after the sleeps,
>>>>>> there's an OOM
>>>>>>
>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have done another test, remember my code that worked, which
>>>>>> previously "warmed" the Thread with an useless request.
>>>>>>
>>>>>> private Runnable uploadSingleDocumentRunnable = new Runnable() {
>>>>>> @Override
>>>>>> public void run() {
>>>>>> try {
>>>>>> getUselessSessionCode();
>>>>>> Thread.sleep(X);
>>>>>> uploadSingleDocument();
>>>>>> } catch ( Exception e ) {
>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>> }
>>>>>> }
>>>>>> };
>>>>>>
>>>>>> I have put a sleep of X between the useless warmup request, and the
>>>>>> real upload request
>>>>>>
>>>>>>
>>>>>> What I noticed is that there is a very different behavior according
>>>>>> to the value of X
>>>>>>
>>>>>>
>>>>>> Under 10 seconds, it seems the stuff is still warm, I can upload the
>>>>>> documents.
>>>>>> Around 10 seconds I get a stack which seems to be "connection closed"
>>>>>> or something
>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>
>>>>>>
>>>>>> The stacks I get for 10 seconds looks like
>>>>>>
>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>
>>>>>> I think I got some other different stacks saying Connection Closed
>>>>>> Remotely or something like that.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So it seems that something is bound to my thread, and it stays bound
>>>>>> to it for about 10 seconds, do you have any idea what it could be?
>>>>>> (My connection timeout setting seems to have no effect on this 10s
>>>>>> threshold)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>
>>>>>>> That is one solution. I'm working out an alternative right now.
>>>>>>> Stay tuned!
>>>>>>>
>>>>>>>
>>>>>>> Anyway it's not a problem, I think the FeedableBodyGenerator.feed()
>>>>>>> method just has to block until a context has been (and ThreadCache
>>>>>>> initialized) to avoid OOM errors
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>>
>>>>>>>> What is very strange is that I tested with/without the same
>>>>>>>> sessionCode with our previous code, the one not using
>>>>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>>>>> Despites the fact it had high memory consumption, it seems work
>>>>>>>> fine to upload multiple documents if allocated with a large heap, and the
>>>>>>>> sessionCode seems to have no effect.
>>>>>>>>
>>>>>>>> On the new impl using the FeedableBodyGenerator, the sessionCode
>>>>>>>> sent as a multipart bodypart seems to have an effect.
>>>>>>>>
>>>>>>>> I have tried to feed the queue before sending the request to AHC,
>>>>>>>> but this leads to this exception (with/without sessionCode switching)
>>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>
>>>>>>>>> By the way, by using a low timeout with the same sessioncode, I
>>>>>>>>> got the following NPE:
>>>>>>>>>
>>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>
>>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>>> httpCtx =
>>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>
>>>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is right, here's a log I have when I use the same session
>>>>>>>>>> code, ie the remote host is blocking the data or something.
>>>>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>>>>
>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268',
>>>>>>>>>> transport: 'socket'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Otherwise, with different session codes, I get the following:
>>>>>>>>>>
>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>> ... and this continues without OOM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So, this seems to be the problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think it would be great to be able to be able to choose the
>>>>>>>>>> queue impl behind that FeedableBodyGenerator, like I suggested in my pull
>>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>> See here:
>>>>>>>>>>
>>>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>
>>>>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in this
>>>>>>>>>> context, and in my case I would probably use a queue of size 1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This would handle the blocking of the feed method, without having
>>>>>>>>>> to use this:
>>>>>>>>>> if (context != null) {
>>>>>>>>>> flushQueue(true);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Or perhaps the feed() method have to wait until a context is set
>>>>>>>>>> in the BodyGenerator ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think it would be more clear if
>>>>>>>>>> the initializeAsynchronousTransfer simply didn't flush the queue but just
>>>>>>>>>> setup the context.
>>>>>>>>>> Then the feed method would block until there's a context set, and
>>>>>>>>>> then flush the queue with blocking behavior.
>>>>>>>>>>
>>>>>>>>>> This is probably the next step, but as we are using AHC for
>>>>>>>>>> async, it would probably be great if that blocking feed() method was called
>>>>>>>>>> in a worker thread instead of our main thread.
>>>>>>>>>> I won't use this but someone who really wants a non-blocking impl
>>>>>>>>>> of performant multipart fileupload would probably need this, or will use an
>>>>>>>>>> ExecutorService for the feeding operations as a workaround.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks again for your reactivity
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>
>>>>>>>>>>> I've integrated this change and it works fine except a little
>>>>>>>>>>> detail.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>>>>> This API requires a "sessionCode" in each request. So there is a
>>>>>>>>>>> multipart StringPart with that SessionCode.
>>>>>>>>>>>
>>>>>>>>>>> We used to have a cache which holds the sessionCode 30min per
>>>>>>>>>>> user so that we do not need to init a new session each time.
>>>>>>>>>>>
>>>>>>>>>>> I had troubles in this very specific case: when I upload 5 docs
>>>>>>>>>>> with the same session code.
>>>>>>>>>>> When I remove the cache and use 5 different session codes, it
>>>>>>>>>>> works fine.
>>>>>>>>>>>
>>>>>>>>>>> I guess the remote service is blocking concurrent uploads with
>>>>>>>>>>> the same session code. I don't know at all.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Where I want to go is that I wouldn't have expected Grizzly to
>>>>>>>>>>> OOM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of each
>>>>>>>>>>> Buffer sent to the FeedableBodyGenerator
>>>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Do you have any idea why is there an OOM with these settings?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Perhaps it is because the feed() method of FeedableBodyGenerator
>>>>>>>>>>> doesn't block until the context is initialized.
>>>>>>>>>>> I guess the initializeAsynchronousTransfer is called only once
>>>>>>>>>>> the connection is established, and perhaps a lot of Buffer are added to the
>>>>>>>>>>> queue...
>>>>>>>>>>>
>>>>>>>>>>> Yes, it's invoked once the request has been dispatched, so if
>>>>>>>>>>> the generator is fed a lot before the request, I could see this happening.
>>>>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But I'm not sure at all because the session code is transmitted
>>>>>>>>>>> as a BodyPart and I get the same problem if i put it as the first or last
>>>>>>>>>>> multipart.
>>>>>>>>>>>
>>>>>>>>>>> It's not a big deal, perhaps I should always use a different
>>>>>>>>>>> session code for concurrent operations but I'd like to be sure that we
>>>>>>>>>>> won't have this issue in production...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>> Good catch. Fixed.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There's a little mistake in the grizzly ahc provider relative
>>>>>>>>>>>> to the write queue size.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>>>
>>>>>>>>>>>> As you can see, the TransportCustomizer is called, and then the
>>>>>>>>>>>> write queue size (among other things) is set to AUTO_SIZE (instead of
>>>>>>>>>>>> previously UNLIMITED)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think the default settings like this AUTO_SIZE attribute
>>>>>>>>>>>> should be set before the customization of the transport, or they would
>>>>>>>>>>>> override the value we customized.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is actually my case, since I can't reproduce my "bug"
>>>>>>>>>>>> which is "high memory consumption", even when using -1 / UNLIMITED in the
>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd rather be
>>>>>>>>>>>> able to tune this parameter during load tests to see the effect.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a 500m
>>>>>>>>>>>>> file with 40m heap size ;)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <ryan.lubke_at_oracle.com> a
>>>>>>>>>>>>> écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm going to be updating the Grizzly provider such that
>>>>>>>>>>>>> AUTO_SIZE (not AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding your tuning question, I would probably set the value
>>>>>>>>>>>>> to AsyncQueueWriter.AUTO_TUNE (this will be four times the socket write
>>>>>>>>>>>>> buffer) and see how that works.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> A question first. With these changes, your memory usage is
>>>>>>>>>>>>> more inline with what you were looking for?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will be
>>>>>>>>>>>>> released (with these new improvements?)
>>>>>>>>>>>>>
>>>>>>>>>>>>> We would like to know if we wait for a release or if we
>>>>>>>>>>>>> install our own temp release on Nexus :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If I let the initial value (-1) it won't block, canWrite
>>>>>>>>>>>>>> always returns true
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value as a
>>>>>>>>>>>>>> config attribute, neither the size of the write buffer you talked about.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So in the end, is there a way with current AHC code to use
>>>>>>>>>>>>>> this "canWrite = false" behavior?
>>>>>>>>>>>>>> If not, can you please provide a way to set this behavior on
>>>>>>>>>>>>>> v1.7.20 ? thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS: does it make sens to use the same number of bytes un the
>>>>>>>>>>>>>> feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you also provide a sample of how you're performing
>>>>>>>>>>>>>>> your feed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> By default, Grizzly will configure the async write queue
>>>>>>>>>>>>>>> length to be four times the write buffer size (which is based off the
>>>>>>>>>>>>>>> socket write buffer).
>>>>>>>>>>>>>>> When this queue exceeds this value, canWrite() will return
>>>>>>>>>>>>>>> false.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When this occurs, you can register a WriteListener to be
>>>>>>>>>>>>>>> notified when the queue length is below the configured max and then
>>>>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>> public void onWritePossible() throws Exception {
>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>>>>> future.get(writeTimeout, TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>>>>> I use AHC default which seems to use SameThreadIOStrategy so
>>>>>>>>>>>>>>> I don't think it's related to the IO strategy.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of 50m,
>>>>>>>>>>>>>>> but I have to put a Thread.sleep(30) between each 100k Buffer send to the
>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The connection with the server is not good here, but in
>>>>>>>>>>>>>>> production it is normally a lot better as far as I know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've tried things
>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd like the Grizzly internals to block when there are too
>>>>>>>>>>>>>>> much pending bytes to send. Is it possible?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PS: I've just been able to send a 500mo file with 100mo
>>>>>>>>>>>>>>> heap, but it needed a sleep of 100ms between each 100k Buffer sent to the
>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner used in
>>>>>>>>>>>>>>>> the SSL filter?
>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>>>>> I don't really understand why messages have to be cloned,
>>>>>>>>>>>>>>>> can I remove this? How?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a heap of
>>>>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In our real use cases we would probably have files under
>>>>>>>>>>>>>>>>> 20mo but we want to reduce the memory consumption because we can have x
>>>>>>>>>>>>>>>>> parallel uploads on the same server according to the user activity.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator reduced the
>>>>>>>>>>>>>>>>> memory footprint or if it's almost like before.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer
>>>>>>>>>>>>>>>>>> allocation is concerned, it's untunable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As to your second question, there is no suggested size.
>>>>>>>>>>>>>>>>>> This is all very application specific.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of 33842
>>>>>>>>>>>>>>>>>> and it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so that it
>>>>>>>>>>>>>>>>>> consumes less memory?
>>>>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed method?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each new buffer
>>>>>>>>>>>>>>>>>>> that is passed to the SSLFilter, we allocate a buffer twice the size in
>>>>>>>>>>>>>>>>>>> order to
>>>>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an
>>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Depending on the socket configurations of both
>>>>>>>>>>>>>>>>>>> endpoints, and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>>>>> be the write queue is becoming too large. We do have a
>>>>>>>>>>>>>>>>>>> way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded here. I
>>>>>>>>>>>>>>>>>>> will see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded like
>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream the body
>>>>>>>>>>>>>>>>>>> or something like that?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull request on
>>>>>>>>>>>>>>>>>>>> AHC tomorrow with a better code using the same Part classes that already
>>>>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder feeds
>>>>>>>>>>>>>>>>>>>> the queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or
>>>>>>>>>>>>>>>>>>>> something, will work on that, it will be better than a thread sleep to slow
>>>>>>>>>>>>>>>>>>>> down the filepart reading
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to tackle
>>>>>>>>>>>>>>>>>>>>> adding something like this today. I'll follow up with something you can
>>>>>>>>>>>>>>>>>>>>> test out.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think I see what I could do, probably something like
>>>>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder =
>>>>>>>>>>>>>>>>>>>>> new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes =
>>>>>>>>>>>>>>>>>>>>> asyncHttpClient
>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>> .execute(new AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to upload a
>>>>>>>>>>>>>>>>>>>>>>> file with AHC / Grizzly in streaming, I mean without loading the whole file
>>>>>>>>>>>>>>>>>>>>>>> bytes in memory.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a byte[]
>>>>>>>>>>>>>>>>>>>>>>> which contans the whole file, so it means that my server can be OOM if too
>>>>>>>>>>>>>>>>>>>>>>> many users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory
>>>>>>>>>>>>>>>>>>>>>>> managers, with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended wto the
>>>>>>>>>>>>>>>>>>>>>>> BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC
>>>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the file
>>>>>>>>>>>>>>>>>>>>>>> so that I could eventually consume only 20mo of heap while uploading a
>>>>>>>>>>>>>>>>>>>>>>> 500mo file?
>>>>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in the
>>>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator. But if
>>>>>>>>>>>>>>>>>>>>>> you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area can be
>>>>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive a file
>>>>>>>>>>>>>>>>>>>>>>> with CXF and have to transmit it to a storage server (like S3). CXF doesn't
>>>>>>>>>>>>>>>>>>>>>>> consume memory bevause it is streaming the large fle uploads to the file
>>>>>>>>>>>>>>>>>>>>>>> system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>