Hi.
Thanks, it seems to work.
I would suggest to throw IOException on the flush method since the feed
method is supposed to be called here
My implementation of flush is:
@Override
public void flush() {
Part[] partsArray = parts.toArray(new Part[parts.size()]);
try ( OutputStream outputStream = createFeedingOutputStream() ) {
Part.sendParts(outputStream,partsArray,multipartBoundary);
} catch (Exception e) {
throw new IllegalStateException("Unable to feed the
FeedableBodyGenerator",e);
}
}
Is this correct? The OutputStream redirects the bytes written to the
feed(Buffer) method
There seems to be some concurrency issue. Because the upload of 1 file
seems fine, but when using multiple threads, I often get the following
stack:
Caused by: java.io.IOException: Stream Closed
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:242)
at
com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
at com.ning.http.multipart.Part.send(Part.java:331)
at com.ning.http.multipart.Part.sendParts(Part.java:397)
This is because the flush() method is called multiple times for the same
request on some cases.
I guess this is not supposed to happen.
What I understand is that the flush() method is supposed to be called only
once.
Using debug logging breakpoints I get the following:
myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10 started
myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11 started
myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12 started
myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13 started
myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14 started
myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15 started
myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16 started
myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17 started
myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18 started
myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19 started
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
Adding handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
*Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
*Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
*Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
*Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
*Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
Completing and removing handshake listener for
com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
As you can see the same HandshakeListener seems to be called multiple times
for the same FeedableBodyGenerator
When this happens, the stack is:
at
com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
~[async-http-client-1.7.20-SNAPSHOT.jar:na]
at
org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
~[async-http-client-1.7.20-SNAPSHOT.jar:na]
at
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
~[grizzly-framework-2.3.5.jar:2.3.5]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
~[grizzly-framework-2.3.5.jar:2.3.5]
So I tried with the following code:
private boolean alreadyFlushed = false;
@Override
public synchronized void flush() {
if ( alreadyFlushed ) {
return;
}
startFeeding();
alreadyFlushed = true;
}
It works fine when upload small files.
But with larger files, I often get TimeoutException stacks for some of the
threads:
Caused by: java.util.concurrent.TimeoutException
at
org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
at org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
at
com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
at
com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
at
com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
Did I miss something?
2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com>
> Committed another small change. Please make sure you're at the latest
> when you build.
>
> -rl
>
>
> Ryan Lubke wrote:
>
> Okay, I've committed another set of refactorings to the
> FeedableBodyGenerator.
>
> For your use case, you should extend FeedableBodyGenerator.SimpleFeeder.
>
> Let me know if you run into issues.
>
>
>
> Sébastien Lorber wrote:
>
> Yes I think it would, so that I can feed the queue at once
>
> One thread will be locked during the feeding for nothing but it's not a
> real problem in my usecase.
>
>
> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>
>> Would having a different listener that will be notified once async
>> transferring has been started work better for you?
>>
>> Something like:
>>
>> onAsyncTransferInitiated() {
>> // invoke your feed method
>> }
>>
>> ?
>>
>>
>>
>> Sébastien Lorber wrote:
>>
>> Unfortunatly I won't be able to use the Feeder non-blocking stuff for
>> now, because of how the multipart request in handled in AHC
>>
>>
>> Here's my feeding method:
>>
>> public void feed() throws IOException {
>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>> try ( OutputStream outputStream = createFeedingOutputStream() ) {
>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>> } catch (Exception e) {
>> throw new IllegalStateException("Unable to feed the
>> FeedableBodyGenerator",e);
>> }
>> }
>>
>>
>> As you can see, the multipart Parts array can only be pushed to the
>> OutputStream, I don't have any way to "pull" the data when the canFeed()
>> method is triggered.
>>
>>
>> But I've seen that there's a com.ning.http.multipart.MultipartBody#read
>> that seems to provide a memory efficient way to pull data from a Multipart
>> body...
>>
>> Should see what I come up with this
>>
>>
>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>
>>> It seems the Feeder is highly recommended but not mandatory so I tried
>>> without.
>>>
>>> With my existing code it seems there is a synchronization problem.
>>>
>>>
>>> The feeding threads get locked to the prematureFeed.get();
>>>
>>> So the Grizzly kernel threads are unable to acquire the lock required to
>>> enter the initializeAsynchronousTransfer method
>>>
>>>
>>>
>>> Will try with an implementation of Feeder
>>>
>>>
>>>
>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>
>>>> Hmmm it seems I have a problem with one of your maven plugins. I'll try
>>>> to bypass it, but for info:
>>>>
>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>> [WARNING]
>>>> [WARNING] Some problems were encountered while building the effective
>>>> settings
>>>> [WARNING] 'profiles.profile[default].repositories.repository.id' must
>>>> be unique but found duplicate repository with id fullsix-maven-repository @
>>>> /home/slorber/.m2/settings.xml
>>>> [WARNING]
>>>> [INFO] Scanning for projects...
>>>> [INFO]
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO]
>>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>>> async-http-client ---
>>>> [INFO]
>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven) @
>>>> async-http-client ---
>>>> [INFO]
>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-versions)
>>>> @ async-http-client ---
>>>> [INFO]
>>>> [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
>>>> async-http-client ---
>>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>> [INFO] skip non existing resourceDirectory
>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>> [INFO]
>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
>>>> async-http-client ---
>>>> [INFO] Compiling 158 source files to
>>>> /home/slorber/Bureau/ahc2/target/classes
>>>> [INFO]
>>>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>> *[INFO] Checking unresolved references to
>>>> org.codehaus.mojo.signature:java15:1.0*
>>>> *[ERROR] Undefined reference:
>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
>>>> *
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 8.747s
>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>> [INFO] Final Memory: 30M/453M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> *[ERROR] Failed to execute goal
>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]*
>>>> [ERROR]
>>>> [ERROR] To see the full stack trace of the errors, re-run Maven with
>>>> the -e switch.
>>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>>> [ERROR]
>>>> [ERROR] For more information about the errors and possible solutions,
>>>> please read the following articles:
>>>> [ERROR] [Help 1]
>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>
>>>>
>>>>
>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>
>>>>> Ok thank you, I'll try to implement that today and will give you my
>>>>> feedback :)
>>>>>
>>>>>
>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>
>>>>>> Okay,
>>>>>>
>>>>>> I've committed my initial changes to the AHC repository. Here's a
>>>>>> summary of the changes:
>>>>>>
>>>>>>
>>>>>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>>> - Don't allow queueing of data before initiateAsyncTransfer has been
>>>>>> invoked. In low memory
>>>>>> heaps, this could lead to an OOM if the source is feeding too fast.
>>>>>> The new behavior is to
>>>>>> block until initiateAsyncTransfer is called, at which time the
>>>>>> blocked thread may proceed with
>>>>>> the feed operation.
>>>>>> - Introduce the concept of a Feeder. Implementations are responsible,
>>>>>> at a high level, for:
>>>>>> + letting the provider know that data is available to be fed without
>>>>>> blocking
>>>>>> + allowing the registration of a callback that the Feeder
>>>>>> implementation may invoke
>>>>>> to signal that more data is available, if it wasn't available at a
>>>>>> previous point in time.
>>>>>> - When using a Feeder with a secure request, the SSL handshake will
>>>>>> be kicked off by the
>>>>>> initiateAsyncTransfer call, but feeding of data will not occur until
>>>>>> the handshake is complete.
>>>>>> This is necessary as the SSLFilter will queue up all writes until the
>>>>>> handshake is complete,
>>>>>> and currently, the buffer isn't tied in with the transport flow
>>>>>> control mechanism.
>>>>>> NOTE: This new SSL behavior is not currently applied when invoking
>>>>>> the feed() method
>>>>>> outside the context of a Feeder. Still need to address that.
>>>>>> - Exposed configuration of the async write queue limit through the
>>>>>> FeedableBodyGenerator.
>>>>>> This is an improvement on using a TransportCustomizer as any
>>>>>> configuration there is
>>>>>> transport-wide, and therefor applied to all Connections. By exposing
>>>>>> it here, each feeder
>>>>>> may have a different byte limit.
>>>>>> - Improved documentation for this class*
>>>>>>
>>>>>> I recommend reading through the javadoc comments in the source [1]
>>>>>> for FeedableBodyGenerator (comments welcome).
>>>>>> Additionally, I would re-work your code to leverage the Feeder
>>>>>> instead of calling feed() directly.
>>>>>>
>>>>>> If you have issues implementing Feeder, do let us know.
>>>>>>
>>>>>> If you have additional questions, again, let us know.
>>>>>>
>>>>>> Thanks,
>>>>>> -rl
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ryan Lubke wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber wrote:
>>>>>>
>>>>>> So in the end I've end up with an implementation that's working for
>>>>>> me.
>>>>>>
>>>>>>
>>>>>> I think there are 2 bugs:
>>>>>>
>>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue if the
>>>>>> initialize(ctx) method is not called fast enough.
>>>>>> This can be solved by using a BlockingQueue of size 1 and the put()
>>>>>> method.
>>>>>>
>>>>>> 2) Once the context is injected, the FeedableBodyGenerator flushes
>>>>>> the queue.
>>>>>> The matter is that if the connection is new, not warmed up by a
>>>>>> previous request, then the SSL handshake is not done yet, and it seems that
>>>>>> the bytes are accumulated in some part of the SSL filter which doesn't
>>>>>> deliver them to the connection until the handshake has completed,
>>>>>> so c.canWrite() continues to return true.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have replaced some part of the FeedableBodyGenerator to test this
>>>>>> and it works pretty fine. See what I have changed:
>>>>>>
>>>>>> 1)
>>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>
>>>>>>
>>>>>> 2)
>>>>>> public void feed(final Buffer buffer, final boolean last) throws
>>>>>> IOException {
>>>>>> try {
>>>>>> queue.put(new BodyPart(buffer, last));
>>>>>> } catch (InterruptedException e) {
>>>>>> throw new RuntimeException(e);
>>>>>> }
>>>>>> queueSize.incrementAndGet();
>>>>>>
>>>>>> if (context != null) {
>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>> flushQueue(true);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> private void
>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>>> try { Thread.sleep(10); } catch ( Exception e ) { throw new
>>>>>> RuntimeException(e); }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> private boolean connectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>> Connection connection = fcc.getConnection();
>>>>>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>>>>>> return sslEngine != null &&
>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>>
>>>>>> We had come to similar conclusions on this end. I'm still working
>>>>>> through testing the idea I mentioned previously (took longer than I
>>>>>> expected - sorry).
>>>>>> I hope to have something for you to test very soon.
>>>>>>
>>>>>> Note that it will be taking the above into account as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>
>>>>>>>
>>>>>>> I have tried to put a while ( context == null ) Thread.sleep but it
>>>>>>> doesn't seem to work, when the context gets injected, after the sleeps,
>>>>>>> there's an OOM
>>>>>>>
>>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have done another test, remember my code that worked, which
>>>>>>> previously "warmed" the Thread with an useless request.
>>>>>>>
>>>>>>> private Runnable uploadSingleDocumentRunnable = new Runnable() {
>>>>>>> @Override
>>>>>>> public void run() {
>>>>>>> try {
>>>>>>> getUselessSessionCode();
>>>>>>> Thread.sleep(X);
>>>>>>> uploadSingleDocument();
>>>>>>> } catch ( Exception e ) {
>>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>>> }
>>>>>>> }
>>>>>>> };
>>>>>>>
>>>>>>> I have put a sleep of X between the useless warmup request, and the
>>>>>>> real upload request
>>>>>>>
>>>>>>>
>>>>>>> What I noticed is that there is a very different behavior according
>>>>>>> to the value of X
>>>>>>>
>>>>>>>
>>>>>>> Under 10 seconds, it seems the stuff is still warm, I can upload the
>>>>>>> documents.
>>>>>>> Around 10 seconds I get a stack which seems to be "connection
>>>>>>> closed" or something
>>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>>
>>>>>>>
>>>>>>> The stacks I get for 10 seconds looks like
>>>>>>>
>>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>
>>>>>>> I think I got some other different stacks saying Connection Closed
>>>>>>> Remotely or something like that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So it seems that something is bound to my thread, and it stays bound
>>>>>>> to it for about 10 seconds, do you have any idea what it could be?
>>>>>>> (My connection timeout setting seems to have no effect on this 10s
>>>>>>> threshold)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>> That is one solution. I'm working out an alternative right now.
>>>>>>>> Stay tuned!
>>>>>>>>
>>>>>>>>
>>>>>>>> Anyway it's not a problem, I think the FeedableBodyGenerator.feed()
>>>>>>>> method just has to block until a context has been (and ThreadCache
>>>>>>>> initialized) to avoid OOM errors
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What is very strange is that I tested with/without the same
>>>>>>>>> sessionCode with our previous code, the one not using
>>>>>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>>>>>> Despites the fact it had high memory consumption, it seems work
>>>>>>>>> fine to upload multiple documents if allocated with a large heap, and the
>>>>>>>>> sessionCode seems to have no effect.
>>>>>>>>>
>>>>>>>>> On the new impl using the FeedableBodyGenerator, the sessionCode
>>>>>>>>> sent as a multipart bodypart seems to have an effect.
>>>>>>>>>
>>>>>>>>> I have tried to feed the queue before sending the request to AHC,
>>>>>>>>> but this leads to this exception (with/without sessionCode switching)
>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>
>>>>>>>>>> By the way, by using a low timeout with the same sessioncode, I
>>>>>>>>>> got the following NPE:
>>>>>>>>>>
>>>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>>>> at
>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>>>> at
>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>> at
>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>> at
>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>
>>>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>>>> httpCtx =
>>>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>
>>>>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is right, here's a log I have when I use the same session
>>>>>>>>>>> code, ie the remote host is blocking the data or something.
>>>>>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>>>>>
>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268',
>>>>>>>>>>> transport: 'socket'
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Otherwise, with different session codes, I get the following:
>>>>>>>>>>>
>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>> ... and this continues without OOM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So, this seems to be the problem.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think it would be great to be able to be able to choose the
>>>>>>>>>>> queue impl behind that FeedableBodyGenerator, like I suggested in my pull
>>>>>>>>>>> request.
>>>>>>>>>>>
>>>>>>>>>>> See here:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>
>>>>>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in this
>>>>>>>>>>> context, and in my case I would probably use a queue of size 1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This would handle the blocking of the feed method, without
>>>>>>>>>>> having to use this:
>>>>>>>>>>> if (context != null) {
>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Or perhaps the feed() method have to wait until a context is set
>>>>>>>>>>> in the BodyGenerator ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think it would be more clear if
>>>>>>>>>>> the initializeAsynchronousTransfer simply didn't flush the queue but just
>>>>>>>>>>> setup the context.
>>>>>>>>>>> Then the feed method would block until there's a context set,
>>>>>>>>>>> and then flush the queue with blocking behavior.
>>>>>>>>>>>
>>>>>>>>>>> This is probably the next step, but as we are using AHC for
>>>>>>>>>>> async, it would probably be great if that blocking feed() method was called
>>>>>>>>>>> in a worker thread instead of our main thread.
>>>>>>>>>>> I won't use this but someone who really wants a non-blocking
>>>>>>>>>>> impl of performant multipart fileupload would probably need this, or will
>>>>>>>>>>> use an ExecutorService for the feeding operations as a workaround.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for your reactivity
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I've integrated this change and it works fine except a little
>>>>>>>>>>>> detail.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>>>>>> This API requires a "sessionCode" in each request. So there is
>>>>>>>>>>>> a multipart StringPart with that SessionCode.
>>>>>>>>>>>>
>>>>>>>>>>>> We used to have a cache which holds the sessionCode 30min per
>>>>>>>>>>>> user so that we do not need to init a new session each time.
>>>>>>>>>>>>
>>>>>>>>>>>> I had troubles in this very specific case: when I upload 5 docs
>>>>>>>>>>>> with the same session code.
>>>>>>>>>>>> When I remove the cache and use 5 different session codes, it
>>>>>>>>>>>> works fine.
>>>>>>>>>>>>
>>>>>>>>>>>> I guess the remote service is blocking concurrent uploads with
>>>>>>>>>>>> the same session code. I don't know at all.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Where I want to go is that I wouldn't have expected Grizzly to
>>>>>>>>>>>> OOM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>> at
>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>> at
>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of each
>>>>>>>>>>>> Buffer sent to the FeedableBodyGenerator
>>>>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have any idea why is there an OOM with these settings?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Perhaps it is because the feed() method of
>>>>>>>>>>>> FeedableBodyGenerator doesn't block until the context is initialized.
>>>>>>>>>>>> I guess the initializeAsynchronousTransfer is called only once
>>>>>>>>>>>> the connection is established, and perhaps a lot of Buffer are added to the
>>>>>>>>>>>> queue...
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, it's invoked once the request has been dispatched, so if
>>>>>>>>>>>> the generator is fed a lot before the request, I could see this happening.
>>>>>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But I'm not sure at all because the session code is transmitted
>>>>>>>>>>>> as a BodyPart and I get the same problem if i put it as the first or last
>>>>>>>>>>>> multipart.
>>>>>>>>>>>>
>>>>>>>>>>>> It's not a big deal, perhaps I should always use a different
>>>>>>>>>>>> session code for concurrent operations but I'd like to be sure that we
>>>>>>>>>>>> won't have this issue in production...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Good catch. Fixed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There's a little mistake in the grizzly ahc provider relative
>>>>>>>>>>>>> to the write queue size.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>>>>
>>>>>>>>>>>>> As you can see, the TransportCustomizer is called, and then
>>>>>>>>>>>>> the write queue size (among other things) is set to AUTO_SIZE (instead of
>>>>>>>>>>>>> previously UNLIMITED)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think the default settings like this AUTO_SIZE attribute
>>>>>>>>>>>>> should be set before the customization of the transport, or they would
>>>>>>>>>>>>> override the value we customized.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is actually my case, since I can't reproduce my "bug"
>>>>>>>>>>>>> which is "high memory consumption", even when using -1 / UNLIMITED in the
>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd rather be
>>>>>>>>>>>>> able to tune this parameter during load tests to see the effect.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a 500m
>>>>>>>>>>>>>> file with 40m heap size ;)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm going to be updating the Grizzly provider such that
>>>>>>>>>>>>>> AUTO_SIZE (not AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding your tuning question, I would probably set the
>>>>>>>>>>>>>> value to AsyncQueueWriter.AUTO_TUNE (this will be four times the socket
>>>>>>>>>>>>>> write buffer) and see how that works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> A question first. With these changes, your memory usage is
>>>>>>>>>>>>>> more inline with what you were looking for?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will be
>>>>>>>>>>>>>> released (with these new improvements?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We would like to know if we wait for a release or if we
>>>>>>>>>>>>>> install our own temp release on Nexus :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If I let the initial value (-1) it won't block, canWrite
>>>>>>>>>>>>>>> always returns true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value as a
>>>>>>>>>>>>>>> config attribute, neither the size of the write buffer you talked about.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So in the end, is there a way with current AHC code to use
>>>>>>>>>>>>>>> this "canWrite = false" behavior?
>>>>>>>>>>>>>>> If not, can you please provide a way to set this behavior on
>>>>>>>>>>>>>>> v1.7.20 ? thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PS: does it make sens to use the same number of bytes un the
>>>>>>>>>>>>>>> feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Could you also provide a sample of how you're performing
>>>>>>>>>>>>>>>> your feed?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By default, Grizzly will configure the async write queue
>>>>>>>>>>>>>>>> length to be four times the write buffer size (which is based off the
>>>>>>>>>>>>>>>> socket write buffer).
>>>>>>>>>>>>>>>> When this queue exceeds this value, canWrite() will return
>>>>>>>>>>>>>>>> false.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When this occurs, you can register a WriteListener to be
>>>>>>>>>>>>>>>> notified when the queue length is below the configured max and then
>>>>>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>> public void onWritePossible() throws Exception {
>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>>>>>> future.get(writeTimeout, TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>>>>>> I use AHC default which seems to use SameThreadIOStrategy
>>>>>>>>>>>>>>>> so I don't think it's related to the IO strategy.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of 50m,
>>>>>>>>>>>>>>>> but I have to put a Thread.sleep(30) between each 100k Buffer send to the
>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The connection with the server is not good here, but in
>>>>>>>>>>>>>>>> production it is normally a lot better as far as I know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've tried things
>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd like the Grizzly internals to block when there are too
>>>>>>>>>>>>>>>> much pending bytes to send. Is it possible?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PS: I've just been able to send a 500mo file with 100mo
>>>>>>>>>>>>>>>> heap, but it needed a sleep of 100ms between each 100k Buffer sent to the
>>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner used in
>>>>>>>>>>>>>>>>> the SSL filter?
>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>>>>>> I don't really understand why messages have to be cloned,
>>>>>>>>>>>>>>>>> can I remove this? How?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a heap
>>>>>>>>>>>>>>>>>> of 400m.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In our real use cases we would probably have files under
>>>>>>>>>>>>>>>>>> 20mo but we want to reduce the memory consumption because we can have x
>>>>>>>>>>>>>>>>>> parallel uploads on the same server according to the user activity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator reduced the
>>>>>>>>>>>>>>>>>> memory footprint or if it's almost like before.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer
>>>>>>>>>>>>>>>>>>> allocation is concerned, it's untunable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As to your second question, there is no suggested size.
>>>>>>>>>>>>>>>>>>> This is all very application specific.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of 33842
>>>>>>>>>>>>>>>>>>> and it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so that
>>>>>>>>>>>>>>>>>>> it consumes less memory?
>>>>>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed method?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each new
>>>>>>>>>>>>>>>>>>>> buffer that is passed to the SSLFilter, we allocate a buffer twice the size
>>>>>>>>>>>>>>>>>>>> in order to
>>>>>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an
>>>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Depending on the socket configurations of both
>>>>>>>>>>>>>>>>>>>> endpoints, and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>>>>>> be the write queue is becoming too large. We do have
>>>>>>>>>>>>>>>>>>>> a way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded here. I
>>>>>>>>>>>>>>>>>>>> will see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded like
>>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream the
>>>>>>>>>>>>>>>>>>>> body or something like that?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull request
>>>>>>>>>>>>>>>>>>>>> on AHC tomorrow with a better code using the same Part classes that already
>>>>>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder feeds
>>>>>>>>>>>>>>>>>>>>> the queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or
>>>>>>>>>>>>>>>>>>>>> something, will work on that, it will be better than a thread sleep to slow
>>>>>>>>>>>>>>>>>>>>> down the filepart reading
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to tackle
>>>>>>>>>>>>>>>>>>>>>> adding something like this today. I'll follow up with something you can
>>>>>>>>>>>>>>>>>>>>>> test out.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think I see what I could do, probably something
>>>>>>>>>>>>>>>>>>>>>> like that:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>>> = new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new
>>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes =
>>>>>>>>>>>>>>>>>>>>>> asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>> .execute(new AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to upload a
>>>>>>>>>>>>>>>>>>>>>>>> file with AHC / Grizzly in streaming, I mean without loading the whole file
>>>>>>>>>>>>>>>>>>>>>>>> bytes in memory.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a byte[]
>>>>>>>>>>>>>>>>>>>>>>>> which contans the whole file, so it means that my server can be OOM if too
>>>>>>>>>>>>>>>>>>>>>>>> many users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory
>>>>>>>>>>>>>>>>>>>>>>>> managers, with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended wto the
>>>>>>>>>>>>>>>>>>>>>>>> BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC
>>>>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the file
>>>>>>>>>>>>>>>>>>>>>>>> so that I could eventually consume only 20mo of heap while uploading a
>>>>>>>>>>>>>>>>>>>>>>>> 500mo file?
>>>>>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in the
>>>>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator. But
>>>>>>>>>>>>>>>>>>>>>>> if you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area can be
>>>>>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive a
>>>>>>>>>>>>>>>>>>>>>>>> file with CXF and have to transmit it to a storage server (like S3). CXF
>>>>>>>>>>>>>>>>>>>>>>>> doesn't consume memory bevause it is streaming the large fle uploads to the
>>>>>>>>>>>>>>>>>>>>>>>> file system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>