Hmmm it seems I have a problem with one of your maven plugins. I'll try to
bypass it, but for info:
➜ ahc2 git:(ahc-1.7.x) mvn clean install
[WARNING]
[WARNING] Some problems were encountered while building the effective
settings
[WARNING] 'profiles.profile[default].repositories.repository.id' must be
unique but found duplicate repository with id fullsix-maven-repository @
/home/slorber/.m2/settings.xml
[WARNING]
[INFO] Scanning for projects...
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
async-http-client ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven) @
async-http-client ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-versions) @
async-http-client ---
[INFO]
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
async-http-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/slorber/Bureau/ahc2/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
async-http-client ---
[INFO] Compiling 158 source files to
/home/slorber/Bureau/ahc2/target/classes
[INFO]
*[INFO] --- animal-sniffer-maven-plugin:1.6:check (check-java-1.5-compat) @
async-http-client ---*
*[INFO] Checking unresolved references to
org.codehaus.mojo.signature:java15:1.0*
*[ERROR] Undefined reference:
java/io/IOException.<init>(Ljava/lang/Throwable;)V in
/home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
*
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 8.747s
[INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
[INFO] Final Memory: 30M/453M
[INFO]
------------------------------------------------------------------------
*[ERROR] Failed to execute goal
org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
(check-java-1.5-compat) on project async-http-client: Signature errors
found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]*
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
> Ok thank you, I'll try to implement that today and will give you my
> feedback :)
>
>
> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>
>> Okay,
>>
>> I've committed my initial changes to the AHC repository. Here's a
>> summary of the changes:
>>
>>
>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>> - Don't allow queueing of data before initiateAsyncTransfer has been
>> invoked. In low memory
>> heaps, this could lead to an OOM if the source is feeding too fast. The
>> new behavior is to
>> block until initiateAsyncTransfer is called, at which time the blocked
>> thread may proceed with
>> the feed operation.
>> - Introduce the concept of a Feeder. Implementations are responsible, at
>> a high level, for:
>> + letting the provider know that data is available to be fed without
>> blocking
>> + allowing the registration of a callback that the Feeder implementation
>> may invoke
>> to signal that more data is available, if it wasn't available at a
>> previous point in time.
>> - When using a Feeder with a secure request, the SSL handshake will be
>> kicked off by the
>> initiateAsyncTransfer call, but feeding of data will not occur until the
>> handshake is complete.
>> This is necessary as the SSLFilter will queue up all writes until the
>> handshake is complete,
>> and currently, the buffer isn't tied in with the transport flow control
>> mechanism.
>> NOTE: This new SSL behavior is not currently applied when invoking the
>> feed() method
>> outside the context of a Feeder. Still need to address that.
>> - Exposed configuration of the async write queue limit through the
>> FeedableBodyGenerator.
>> This is an improvement on using a TransportCustomizer as any
>> configuration there is
>> transport-wide, and therefor applied to all Connections. By exposing it
>> here, each feeder
>> may have a different byte limit.
>> - Improved documentation for this class*
>>
>> I recommend reading through the javadoc comments in the source [1] for
>> FeedableBodyGenerator (comments welcome).
>> Additionally, I would re-work your code to leverage the Feeder instead of
>> calling feed() directly.
>>
>> If you have issues implementing Feeder, do let us know.
>>
>> If you have additional questions, again, let us know.
>>
>> Thanks,
>> -rl
>>
>> [1]
>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>
>>
>>
>> Ryan Lubke wrote:
>>
>>
>>
>> Sébastien Lorber wrote:
>>
>> So in the end I've end up with an implementation that's working for me.
>>
>>
>> I think there are 2 bugs:
>>
>> 1) The bytes can accumulate in the FeedableBodyGenerator queue if the
>> initialize(ctx) method is not called fast enough.
>> This can be solved by using a BlockingQueue of size 1 and the put()
>> method.
>>
>> 2) Once the context is injected, the FeedableBodyGenerator flushes the
>> queue.
>> The matter is that if the connection is new, not warmed up by a previous
>> request, then the SSL handshake is not done yet, and it seems that the
>> bytes are accumulated in some part of the SSL filter which doesn't deliver
>> them to the connection until the handshake has completed, so c.canWrite()
>> continues to return true.
>>
>>
>>
>>
>> I have replaced some part of the FeedableBodyGenerator to test this and
>> it works pretty fine. See what I have changed:
>>
>> 1)
>> private final BlockingQueue<BodyPart> queue = new
>> LinkedBlockingQueue<BodyPart>(1);
>>
>>
>> 2)
>> public void feed(final Buffer buffer, final boolean last) throws
>> IOException {
>> try {
>> queue.put(new BodyPart(buffer, last));
>> } catch (InterruptedException e) {
>> throw new RuntimeException(e);
>> }
>> queueSize.incrementAndGet();
>>
>> if (context != null) {
>> blockUntilConnectionIsReadyToWrite(context);
>> flushQueue(true);
>> }
>> }
>>
>> private void blockUntilConnectionIsReadyToWrite(FilterChainContext
>> fcc) {
>> if ( !connectionIsReadyToWrite(fcc) ) {
>> while ( !connectionIsReadyToWrite(fcc) ) {
>> try { Thread.sleep(10); } catch ( Exception e ) { throw new
>> RuntimeException(e); }
>> }
>> }
>> }
>>
>> private boolean connectionIsReadyToWrite(FilterChainContext fcc) {
>> Connection connection = fcc.getConnection();
>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>> return sslEngine != null && !SSLUtils.isHandshaking(sslEngine);
>> }
>>
>>
>>
>> What do you think?
>>
>>
>> We had come to similar conclusions on this end. I'm still working
>> through testing the idea I mentioned previously (took longer than I
>> expected - sorry).
>> I hope to have something for you to test very soon.
>>
>> Note that it will be taking the above into account as well.
>>
>>
>>
>>
>>
>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>
>>>
>>> I have tried to put a while ( context == null ) Thread.sleep but it
>>> doesn't seem to work, when the context gets injected, after the sleeps,
>>> there's an OOM
>>>
>>> So I hope you'll have more success with your alternative :)
>>>
>>>
>>>
>>> I have done another test, remember my code that worked, which previously
>>> "warmed" the Thread with an useless request.
>>>
>>> private Runnable uploadSingleDocumentRunnable = new Runnable() {
>>> @Override
>>> public void run() {
>>> try {
>>> getUselessSessionCode();
>>> Thread.sleep(X);
>>> uploadSingleDocument();
>>> } catch ( Exception e ) {
>>> throw new RuntimeException("file upload failed",e);
>>> }
>>> }
>>> };
>>>
>>> I have put a sleep of X between the useless warmup request, and the real
>>> upload request
>>>
>>>
>>> What I noticed is that there is a very different behavior according to
>>> the value of X
>>>
>>>
>>> Under 10 seconds, it seems the stuff is still warm, I can upload the
>>> documents.
>>> Around 10 seconds I get a stack which seems to be "connection closed" or
>>> something
>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>
>>>
>>> The stacks I get for 10 seconds looks like
>>>
>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>
>>> I think I got some other different stacks saying Connection Closed
>>> Remotely or something like that.
>>>
>>>
>>>
>>>
>>> So it seems that something is bound to my thread, and it stays bound to
>>> it for about 10 seconds, do you have any idea what it could be?
>>> (My connection timeout setting seems to have no effect on this 10s
>>> threshold)
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>
>>>> That is one solution. I'm working out an alternative right now. Stay
>>>> tuned!
>>>>
>>>>
>>>> Anyway it's not a problem, I think the FeedableBodyGenerator.feed()
>>>> method just has to block until a context has been (and ThreadCache
>>>> initialized) to avoid OOM errors
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>
>>>>>
>>>>> What is very strange is that I tested with/without the same
>>>>> sessionCode with our previous code, the one not using
>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>> Despites the fact it had high memory consumption, it seems work fine
>>>>> to upload multiple documents if allocated with a large heap, and the
>>>>> sessionCode seems to have no effect.
>>>>>
>>>>> On the new impl using the FeedableBodyGenerator, the sessionCode sent
>>>>> as a multipart bodypart seems to have an effect.
>>>>>
>>>>> I have tried to feed the queue before sending the request to AHC, but
>>>>> this leads to this exception (with/without sessionCode switching)
>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>> at
>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>> at
>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>> at
>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>
>>>>>> By the way, by using a low timeout with the same sessioncode, I got
>>>>>> the following NPE:
>>>>>>
>>>>>> Caused by: java.lang.NullPointerException
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>
>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext httpCtx =
>>>>>> getHttpTransactionContext(c);
>>>>>> httpCtx.abort(e.getCause());
>>>>>>
>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>
>>>>>>
>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is right, here's a log I have when I use the same session code,
>>>>>>> ie the remote host is blocking the data or something.
>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>
>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268',
>>>>>>> transport: 'socket'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Otherwise, with different session codes, I get the following:
>>>>>>>
>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> ... and this continues without OOM
>>>>>>>
>>>>>>>
>>>>>>> So, this seems to be the problem.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think it would be great to be able to be able to choose the queue
>>>>>>> impl behind that FeedableBodyGenerator, like I suggested in my pull request.
>>>>>>>
>>>>>>> See here:
>>>>>>>
>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>
>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in this context,
>>>>>>> and in my case I would probably use a queue of size 1
>>>>>>>
>>>>>>>
>>>>>>> This would handle the blocking of the feed method, without having to
>>>>>>> use this:
>>>>>>> if (context != null) {
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Or perhaps the feed() method have to wait until a context is set in
>>>>>>> the BodyGenerator ?
>>>>>>>
>>>>>>>
>>>>>>> I think it would be more clear if the initializeAsynchronousTransfer
>>>>>>> simply didn't flush the queue but just setup the context.
>>>>>>> Then the feed method would block until there's a context set, and
>>>>>>> then flush the queue with blocking behavior.
>>>>>>>
>>>>>>> This is probably the next step, but as we are using AHC for async,
>>>>>>> it would probably be great if that blocking feed() method was called in a
>>>>>>> worker thread instead of our main thread.
>>>>>>> I won't use this but someone who really wants a non-blocking impl of
>>>>>>> performant multipart fileupload would probably need this, or will use an
>>>>>>> ExecutorService for the feeding operations as a workaround.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks again for your reactivity
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>
>>>>>>>> I've integrated this change and it works fine except a little
>>>>>>>> detail.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>> This API requires a "sessionCode" in each request. So there is a
>>>>>>>> multipart StringPart with that SessionCode.
>>>>>>>>
>>>>>>>> We used to have a cache which holds the sessionCode 30min per user
>>>>>>>> so that we do not need to init a new session each time.
>>>>>>>>
>>>>>>>> I had troubles in this very specific case: when I upload 5 docs
>>>>>>>> with the same session code.
>>>>>>>> When I remove the cache and use 5 different session codes, it works
>>>>>>>> fine.
>>>>>>>>
>>>>>>>> I guess the remote service is blocking concurrent uploads with the
>>>>>>>> same session code. I don't know at all.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Where I want to go is that I wouldn't have expected Grizzly to OOM
>>>>>>>>
>>>>>>>>
>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of each
>>>>>>>> Buffer sent to the FeedableBodyGenerator
>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have any idea why is there an OOM with these settings?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Perhaps it is because the feed() method of FeedableBodyGenerator
>>>>>>>> doesn't block until the context is initialized.
>>>>>>>> I guess the initializeAsynchronousTransfer is called only once the
>>>>>>>> connection is established, and perhaps a lot of Buffer are added to the
>>>>>>>> queue...
>>>>>>>>
>>>>>>>> Yes, it's invoked once the request has been dispatched, so if the
>>>>>>>> generator is fed a lot before the request, I could see this happening.
>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>
>>>>>>>>
>>>>>>>> But I'm not sure at all because the session code is transmitted as
>>>>>>>> a BodyPart and I get the same problem if i put it as the first or last
>>>>>>>> multipart.
>>>>>>>>
>>>>>>>> It's not a big deal, perhaps I should always use a different
>>>>>>>> session code for concurrent operations but I'd like to be sure that we
>>>>>>>> won't have this issue in production...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>
>>>>>>>>> Good catch. Fixed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There's a little mistake in the grizzly ahc provider relative to
>>>>>>>>> the write queue size.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>
>>>>>>>>> As you can see, the TransportCustomizer is called, and then the
>>>>>>>>> write queue size (among other things) is set to AUTO_SIZE (instead of
>>>>>>>>> previously UNLIMITED)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think the default settings like this AUTO_SIZE attribute should
>>>>>>>>> be set before the customization of the transport, or they would override
>>>>>>>>> the value we customized.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is actually my case, since I can't reproduce my "bug" which
>>>>>>>>> is "high memory consumption", even when using -1 / UNLIMITED in the
>>>>>>>>> TransportCustomizer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd rather be able
>>>>>>>>> to tune this parameter during load tests to see the effect.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>
>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a 500m file
>>>>>>>>>> with 40m heap size ;)
>>>>>>>>>>
>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>
>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <ryan.lubke_at_oracle.com> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>> I'm going to be updating the Grizzly provider such that AUTO_SIZE
>>>>>>>>>> (not AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>
>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>
>>>>>>>>>> Regarding your tuning question, I would probably set the value to
>>>>>>>>>> AsyncQueueWriter.AUTO_TUNE (this will be four times the socket write
>>>>>>>>>> buffer) and see how that works.
>>>>>>>>>>
>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>
>>>>>>>>>> A question first. With these changes, your memory usage is more
>>>>>>>>>> inline with what you were looking for?
>>>>>>>>>>
>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>
>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will be released
>>>>>>>>>> (with these new improvements?)
>>>>>>>>>>
>>>>>>>>>> We would like to know if we wait for a release or if we install
>>>>>>>>>> our own temp release on Nexus :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>> ->
>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If I let the initial value (-1) it won't block, canWrite always
>>>>>>>>>>> returns true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value as a config
>>>>>>>>>>> attribute, neither the size of the write buffer you talked about.
>>>>>>>>>>>
>>>>>>>>>>> So in the end, is there a way with current AHC code to use this
>>>>>>>>>>> "canWrite = false" behavior?
>>>>>>>>>>> If not, can you please provide a way to set this behavior on
>>>>>>>>>>> v1.7.20 ? thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> PS: does it make sens to use the same number of bytes un the
>>>>>>>>>>> feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>
>>>>>>>>>>>> Could you also provide a sample of how you're performing your
>>>>>>>>>>>> feed?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> -rl
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>
>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>
>>>>>>>>>>>> By default, Grizzly will configure the async write queue length
>>>>>>>>>>>> to be four times the write buffer size (which is based off the socket write
>>>>>>>>>>>> buffer).
>>>>>>>>>>>> When this queue exceeds this value, canWrite() will return
>>>>>>>>>>>> false.
>>>>>>>>>>>>
>>>>>>>>>>>> When this occurs, you can register a WriteListener to be
>>>>>>>>>>>> notified when the queue length is below the configured max and then
>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>
>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>>>>>>
>>>>>>>>>>>> @Override
>>>>>>>>>>>> public void onWritePossible() throws Exception {
>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> @Override
>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>> }
>>>>>>>>>>>> });
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>> future.get(writeTimeout, TimeUnit.MILLISECONDS);
>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>> I use AHC default which seems to use SameThreadIOStrategy so I
>>>>>>>>>>>> don't think it's related to the IO strategy.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of 50m, but I
>>>>>>>>>>>> have to put a Thread.sleep(30) between each 100k Buffer send to the
>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>
>>>>>>>>>>>> The connection with the server is not good here, but in
>>>>>>>>>>>> production it is normally a lot better as far as I know.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've tried things
>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like the Grizzly internals to block when there are too much
>>>>>>>>>>>> pending bytes to send. Is it possible?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> PS: I've just been able to send a 500mo file with 100mo heap,
>>>>>>>>>>>> but it needed a sleep of 100ms between each 100k Buffer sent to the
>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner used in the
>>>>>>>>>>>>> SSL filter?
>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>> I don't really understand why messages have to be cloned, can
>>>>>>>>>>>>> I remove this? How?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a heap of
>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In our real use cases we would probably have files under 20mo
>>>>>>>>>>>>>> but we want to reduce the memory consumption because we can have x parallel
>>>>>>>>>>>>>> uploads on the same server according to the user activity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator reduced the
>>>>>>>>>>>>>> memory footprint or if it's almost like before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer allocation
>>>>>>>>>>>>>>> is concerned, it's untunable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As to your second question, there is no suggested size.
>>>>>>>>>>>>>>> This is all very application specific.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of 33842 and
>>>>>>>>>>>>>>> it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so that it
>>>>>>>>>>>>>>> consumes less memory?
>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed method?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each new buffer
>>>>>>>>>>>>>>>> that is passed to the SSLFilter, we allocate a buffer twice the size in
>>>>>>>>>>>>>>>> order to
>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an increase.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Depending on the socket configurations of both endpoints,
>>>>>>>>>>>>>>>> and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>> be the write queue is becoming too large. We do have a
>>>>>>>>>>>>>>>> way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded here. I will
>>>>>>>>>>>>>>>> see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded like before.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream the body or
>>>>>>>>>>>>>>>> something like that?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull request on
>>>>>>>>>>>>>>>>> AHC tomorrow with a better code using the same Part classes that already
>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder feeds the
>>>>>>>>>>>>>>>>> queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or something,
>>>>>>>>>>>>>>>>> will work on that, it will be better than a thread sleep to slow down the
>>>>>>>>>>>>>>>>> filepart reading
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to tackle adding
>>>>>>>>>>>>>>>>>> something like this today. I'll follow up with something you can test out.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think I see what I could do, probably something like
>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder =
>>>>>>>>>>>>>>>>>> new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new RequestBuilder("POST")
>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes = asyncHttpClient
>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>> .execute(new AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to upload a file
>>>>>>>>>>>>>>>>>>>> with AHC / Grizzly in streaming, I mean without loading the whole file
>>>>>>>>>>>>>>>>>>>> bytes in memory.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a byte[] which
>>>>>>>>>>>>>>>>>>>> contans the whole file, so it means that my server can be OOM if too many
>>>>>>>>>>>>>>>>>>>> users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory managers,
>>>>>>>>>>>>>>>>>>>> with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended wto the
>>>>>>>>>>>>>>>>>>>> BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC integration:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the file so
>>>>>>>>>>>>>>>>>>>> that I could eventually consume only 20mo of heap while uploading a 500mo
>>>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in the
>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator. But if
>>>>>>>>>>>>>>>>>>> you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area can be
>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive a file
>>>>>>>>>>>>>>>>>>>> with CXF and have to transmit it to a storage server (like S3). CXF doesn't
>>>>>>>>>>>>>>>>>>>> consume memory bevause it is streaming the large fle uploads to the file
>>>>>>>>>>>>>>>>>>>> system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>