So it seems I can reproduce this in local.
Disabling the ssl connection pooling solve the problem.
We didn't have this problems with v1.7.7 we previously used.
I just tested with the last commits of 1.7.20-SNAPSHOT, the problem is
still here.
The problem doesn't appear on normal AHC requests, neither on multipart
file upload when I do not use the FeedableBodyGenerator so it seems this is
the new behavior of FeedableBodyGenerator that has a problem with the ssl
connection pooling. Will try to investigate this.
By the way, your last commits about selector/worker threads seems ok
(didn't notice any problem with the ssl pool disabled)
2013/9/18 Sébastien Lorber <lorber.sebastien_at_gmail.com>
> Hi
>
> Unfortunatly it seems there's another problem.
> On our test environment I often get in the logs:
>
> Caught internal server errorjava.io.IOException: Maximum pooled
> connections exceeded
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.cleanup(GrizzlyAsyncHttpProvider.java:1415)
> ~[async-http-client-1.7.20-2846817.jar:na]
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpPacketParsed(GrizzlyAsyncHttpProvider.java:1366)
> ~[async-http-client-1.7.20-2846817.jar:na]
> at
> org.glassfish.grizzly.http.HttpCodecFilter.decodeWithTransferEncoding(HttpCodecFilter.java:1176)
> ~[grizzly-http-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:516)
> ~[grizzly-http-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:161)
> ~[grizzly-http-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> Wrapped by: java.util.concurrent.ExecutionException: java.io.IOException:
> Maximum pooled connections exceeded
> at
> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:359)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:265)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> com.ning.http.client.providers.grizzly.GrizzlyResponseFuture.get(GrizzlyResponseFuture.java:165)
> ~[async-http-client-1.7.20-2846817.jar:na]
>
>
>
>
> boolean canReturnConnection(final Connection c) {
>
> return (DO_NOT_CACHE.get(c) != null ||
> pool.canCacheConnection());
>
> }
>
>
> private static HttpTransactionContext cleanup(final
> FilterChainContext ctx,
> final
> GrizzlyAsyncHttpProvider provider) {
>
> final Connection c = ctx.getConnection();
> final HttpTransactionContext context =
> provider.getHttpTransactionContext(c);
> context.provider.setHttpTransactionContext(c, null);
> if
> (!context.provider.connectionManager.canReturnConnection(c)) {
> context.abort(new IOException("Maximum pooled connections
> exceeded"));
> } else {
> if
> (!context.provider.connectionManager.returnConnection(context.request, c)) {
> ctx.getConnection().close();
> }
> }
>
> return context;
>
> }
>
>
>
> We use:
>
> Key=[properties.httpclient.allowSslConnectionPool] Value=[true]
> Key=[properties.httpclient.maximumConnectionsPerHost] Value=[20]
> Key=[properties.httpclient.maximumConnectionsTotal] Value=[30]
> Key=[properties.httpclient.timeout.connection] Value=[10000]
> Key=[properties.httpclient.timeout.request] Value=[30000]
>
>
> In our logs, I can see there are 30 times this log:
>
> Remote closed connection (TCPNIOConnection{localSocketAddress={/
> 172.16.104.160:55488}, peerSocketAddress={host/172.16.4.100:443}}).
> Removing from cache
>
> and then it seems no connection is never available and the app is blocked.
>
> I'll try tomorrow to disable or increase the pool size, but if you have
> any idea of the problem please let me know.
> We do not run load tests, these are simple functional tests with nearly no
> concurrency.
>
>
>
> Thanks
>
>
>
>
> 2013/9/17 Ryan Lubke <ryan.lubke_at_oracle.com>
>
>> This is good to hear. Thanks for all the feedback and working with us to
>> nail this down.
>>
>>
>> Sébastien Lorber wrote:
>>
>> Thanks,
>>
>> We already installed the previous snapshot in our nexus because it works
>> fine and I'm working on something else now but I'll give you a feedback
>> soon to see if this still works fine.
>> For us it doesn't really mater which thread is using since we do not use
>> Future in our applications for this case.
>>
>>
>> Btw I've been able deploy a main with my code running concurrent uploads
>> on a server which has a better connectivity with the remote API and it
>> seems I can upload up to 150 concurrent files of 10mb with a heap of 250mo
>> in 30 seconds and a throughput near 50mb/s
>>
>> I don't really know the infrastructure on which my code was deployed but
>> I suspect Grizzly is not the bottleneck anymore :)
>> When I have more than 150 concurrent uploads the remote endpoint seems to
>> timeout under the load.
>>
>>
>>
>>
>> 2013/9/16 Ryan Lubke <ryan.lubke_at_oracle.com>
>>
>>>
>>>
>>> Ryan Lubke wrote:
>>>
>>>
>>>
>>> Sébastien Lorber wrote:
>>>
>>>
>>>
>>>
>>> I noticed something strange.
>>> On the FileInputStream I have, I've added a log on the close() of the
>>> stream which is called once the whole file has been read to be sent to the
>>> feeder.
>>>
>>> 1) If I perform a request (ie my session init request in the previous
>>> discussions) before doing my multipart upload, the thread that does execute
>>> the feeding is the thread that fires the request, and not a Grizzly worker.
>>> [Thread-30] INFO Will close file stream java.io.FileInputStream_at_27fe4315after having read 1440304 bytes
>>>
>>>
>>> 2) If I don't do any request before firing the multipart upload, the
>>> thread that does the feeding is a Grizzly threadpool worker thread:
>>> [Grizzly(22)] INFO Will close file stream
>>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>>
>>>
>>> Is this normal? I would expect a worker thread to always be used, and
>>> the main applicative thread that performs the request to never be blocking.
>>> (but it's not such an issue for me, we don't have a reactive non-blocking
>>> app anyway)
>>>
>>> It's currently expected behavior. Will need to re-evaluate this based
>>> on the new semantics of the FeedableBodyGenerator.
>>>
>>> I've committed a change for this. If, upon testing, you find there is
>>> still an issue, please let us know.
>>>
>>>
>>>
>>> On the 1st case, here's the stacktrace when the feed method is called:
>>>
>>> * at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)
>>> *
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
>>> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>>> at
>>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
>>> * at
>>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)
>>> *
>>>
>>> So it seems this happen when the handshake has already been done when *
>>> initializeAsynchronousTransfer* is called, so that we do not go through
>>> the HandshakeListener
>>>
>>>
>>> Notice that this case seems to also happen in a few threads on the 2nd
>>> case, but most of the threads are Grizzly workers.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>
>>>>
>>>>
>>>> Hi.
>>>>
>>>>
>>>> Thanks, it seems to work.
>>>>
>>>> I would suggest to throw IOException on the flush method since the feed
>>>> method is supposed to be called here
>>>>
>>>>
>>>> My implementation of flush is:
>>>>
>>>> @Override
>>>> public void flush() {
>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>> try ( OutputStream outputStream = createFeedingOutputStream() ) {
>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>> } catch (Exception e) {
>>>> throw new IllegalStateException("Unable to feed the
>>>> FeedableBodyGenerator",e);
>>>> }
>>>> }
>>>>
>>>> Is this correct? The OutputStream redirects the bytes written to the
>>>> feed(Buffer) method
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> There seems to be some concurrency issue. Because the upload of 1 file
>>>> seems fine, but when using multiple threads, I often get the following
>>>> stack:
>>>> Caused by: java.io.IOException: Stream Closed
>>>> at java.io.FileInputStream.readBytes(Native Method)
>>>> at java.io.FileInputStream.read(FileInputStream.java:242)
>>>> at
>>>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>
>>>>
>>>> This is because the flush() method is called multiple times for the
>>>> same request on some cases.
>>>> I guess this is not supposed to happen.
>>>> What I understand is that the flush() method is supposed to be called
>>>> only once.
>>>>
>>>>
>>>> Using debug logging breakpoints I get the following:
>>>>
>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10 started
>>>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11 started
>>>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12 started
>>>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13 started
>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14 started
>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15 started
>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16 started
>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17 started
>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18 started
>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19 started
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>>>> Adding handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>> *Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>>>> *Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>>>> *Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>>>> *Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>>>> *Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>> Completing and removing handshake listener for
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>>>
>>>>
>>>> As you can see the same HandshakeListener seems to be called multiple
>>>> times for the same FeedableBodyGenerator
>>>>
>>>> When this happens, the stack is:
>>>>
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>> at
>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>
>>>>
>>>>
>>>>
>>>> So I tried with the following code:
>>>>
>>>> private boolean alreadyFlushed = false;
>>>> @Override
>>>> public synchronized void flush() {
>>>> if ( alreadyFlushed ) {
>>>> return;
>>>> }
>>>> startFeeding();
>>>> alreadyFlushed = true;
>>>> }
>>>>
>>>> It works fine when upload small files.
>>>>
>>>> But with larger files, I often get TimeoutException stacks for some of
>>>> the threads:
>>>>
>>>> Caused by: java.util.concurrent.TimeoutException
>>>> at
>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>> at
>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>>>>
>>>>
>>>>
>>>>
>>>> Did I miss something?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>
>>>>> Committed another small change. Please make sure you're at the latest
>>>>> when you build.
>>>>>
>>>>> -rl
>>>>>
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>
>>>>> Okay, I've committed another set of refactorings to the
>>>>> FeedableBodyGenerator.
>>>>>
>>>>> For your use case, you should extend
>>>>> FeedableBodyGenerator.SimpleFeeder.
>>>>>
>>>>> Let me know if you run into issues.
>>>>>
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>
>>>>> Yes I think it would, so that I can feed the queue at once
>>>>>
>>>>> One thread will be locked during the feeding for nothing but it's not
>>>>> a real problem in my usecase.
>>>>>
>>>>>
>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>
>>>>>> Would having a different listener that will be notified once async
>>>>>> transferring has been started work better for you?
>>>>>>
>>>>>> Something like:
>>>>>>
>>>>>> onAsyncTransferInitiated() {
>>>>>> // invoke your feed method
>>>>>> }
>>>>>>
>>>>>> ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber wrote:
>>>>>>
>>>>>> Unfortunatly I won't be able to use the Feeder non-blocking stuff for
>>>>>> now, because of how the multipart request in handled in AHC
>>>>>>
>>>>>>
>>>>>> Here's my feeding method:
>>>>>>
>>>>>> public void feed() throws IOException {
>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>> try ( OutputStream outputStream = createFeedingOutputStream() ) {
>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>> } catch (Exception e) {
>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>> FeedableBodyGenerator",e);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> As you can see, the multipart Parts array can only be pushed to the
>>>>>> OutputStream, I don't have any way to "pull" the data when the canFeed()
>>>>>> method is triggered.
>>>>>>
>>>>>>
>>>>>> But I've seen that there's
>>>>>> a com.ning.http.multipart.MultipartBody#read that seems to provide a memory
>>>>>> efficient way to pull data from a Multipart body...
>>>>>>
>>>>>> Should see what I come up with this
>>>>>>
>>>>>>
>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>
>>>>>>> It seems the Feeder is highly recommended but not mandatory so I
>>>>>>> tried without.
>>>>>>>
>>>>>>> With my existing code it seems there is a synchronization problem.
>>>>>>>
>>>>>>>
>>>>>>> The feeding threads get locked to the prematureFeed.get();
>>>>>>>
>>>>>>> So the Grizzly kernel threads are unable to acquire the lock
>>>>>>> required to enter the initializeAsynchronousTransfer method
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Will try with an implementation of Feeder
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>> Hmmm it seems I have a problem with one of your maven plugins. I'll
>>>>>>>> try to bypass it, but for info:
>>>>>>>>
>>>>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>>>>> [WARNING]
>>>>>>>> [WARNING] Some problems were encountered while building the
>>>>>>>> effective settings
>>>>>>>> [WARNING] 'profiles.profile[default].repositories.repository.id'
>>>>>>>> must be unique but found duplicate repository with id
>>>>>>>> fullsix-maven-repository @ /home/slorber/.m2/settings.xml
>>>>>>>> [WARNING]
>>>>>>>> [INFO] Scanning for projects...
>>>>>>>> [INFO]
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO]
>>>>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>>>>>>> async-http-client ---
>>>>>>>> [INFO]
>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven)
>>>>>>>> @ async-http-client ---
>>>>>>>> [INFO]
>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>>>>>>>> (enforce-versions) @ async-http-client ---
>>>>>>>> [INFO]
>>>>>>>> [INFO] --- maven-resources-plugin:2.4.3:resources
>>>>>>>> (default-resources) @ async-http-client ---
>>>>>>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>>>>> [INFO]
>>>>>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
>>>>>>>> async-http-client ---
>>>>>>>> [INFO] Compiling 158 source files to
>>>>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>>>>> [INFO]
>>>>>>>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>>>>>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>>>>>> *[INFO] Checking unresolved references to
>>>>>>>> org.codehaus.mojo.signature:java15:1.0*
>>>>>>>> *[ERROR] Undefined reference:
>>>>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>>>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
>>>>>>>> *
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 8.747s
>>>>>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>>>>>> [INFO] Final Memory: 30M/453M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> *[ERROR] Failed to execute goal
>>>>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>>>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>>>>>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]
>>>>>>>> *
>>>>>>>> [ERROR]
>>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
>>>>>>>> with the -e switch.
>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
>>>>>>>> logging.
>>>>>>>> [ERROR]
>>>>>>>> [ERROR] For more information about the errors and possible
>>>>>>>> solutions, please read the following articles:
>>>>>>>> [ERROR] [Help 1]
>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>
>>>>>>>>> Ok thank you, I'll try to implement that today and will give you
>>>>>>>>> my feedback :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>
>>>>>>>>>> Okay,
>>>>>>>>>>
>>>>>>>>>> I've committed my initial changes to the AHC repository. Here's
>>>>>>>>>> a summary of the changes:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>>>>>>> - Don't allow queueing of data before initiateAsyncTransfer has
>>>>>>>>>> been invoked. In low memory
>>>>>>>>>> heaps, this could lead to an OOM if the source is feeding too
>>>>>>>>>> fast. The new behavior is to
>>>>>>>>>> block until initiateAsyncTransfer is called, at which time the
>>>>>>>>>> blocked thread may proceed with
>>>>>>>>>> the feed operation.
>>>>>>>>>> - Introduce the concept of a Feeder. Implementations are
>>>>>>>>>> responsible, at a high level, for:
>>>>>>>>>> + letting the provider know that data is available to be fed
>>>>>>>>>> without blocking
>>>>>>>>>> + allowing the registration of a callback that the Feeder
>>>>>>>>>> implementation may invoke
>>>>>>>>>> to signal that more data is available, if it wasn't available at
>>>>>>>>>> a previous point in time.
>>>>>>>>>> - When using a Feeder with a secure request, the SSL handshake
>>>>>>>>>> will be kicked off by the
>>>>>>>>>> initiateAsyncTransfer call, but feeding of data will not occur
>>>>>>>>>> until the handshake is complete.
>>>>>>>>>> This is necessary as the SSLFilter will queue up all writes until
>>>>>>>>>> the handshake is complete,
>>>>>>>>>> and currently, the buffer isn't tied in with the transport flow
>>>>>>>>>> control mechanism.
>>>>>>>>>> NOTE: This new SSL behavior is not currently applied when
>>>>>>>>>> invoking the feed() method
>>>>>>>>>> outside the context of a Feeder. Still need to address that.
>>>>>>>>>> - Exposed configuration of the async write queue limit through
>>>>>>>>>> the FeedableBodyGenerator.
>>>>>>>>>> This is an improvement on using a TransportCustomizer as any
>>>>>>>>>> configuration there is
>>>>>>>>>> transport-wide, and therefor applied to all Connections. By
>>>>>>>>>> exposing it here, each feeder
>>>>>>>>>> may have a different byte limit.
>>>>>>>>>> - Improved documentation for this class*
>>>>>>>>>>
>>>>>>>>>> I recommend reading through the javadoc comments in the source
>>>>>>>>>> [1] for FeedableBodyGenerator (comments welcome).
>>>>>>>>>> Additionally, I would re-work your code to leverage the Feeder
>>>>>>>>>> instead of calling feed() directly.
>>>>>>>>>>
>>>>>>>>>> If you have issues implementing Feeder, do let us know.
>>>>>>>>>>
>>>>>>>>>> If you have additional questions, again, let us know.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> -rl
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>
>>>>>>>>>> So in the end I've end up with an implementation that's working
>>>>>>>>>> for me.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think there are 2 bugs:
>>>>>>>>>>
>>>>>>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue if
>>>>>>>>>> the initialize(ctx) method is not called fast enough.
>>>>>>>>>> This can be solved by using a BlockingQueue of size 1 and the
>>>>>>>>>> put() method.
>>>>>>>>>>
>>>>>>>>>> 2) Once the context is injected, the FeedableBodyGenerator
>>>>>>>>>> flushes the queue.
>>>>>>>>>> The matter is that if the connection is new, not warmed up by a
>>>>>>>>>> previous request, then the SSL handshake is not done yet, and it seems that
>>>>>>>>>> the bytes are accumulated in some part of the SSL filter which doesn't
>>>>>>>>>> deliver them to the connection until the handshake has completed,
>>>>>>>>>> so c.canWrite() continues to return true.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have replaced some part of the FeedableBodyGenerator to test
>>>>>>>>>> this and it works pretty fine. See what I have changed:
>>>>>>>>>>
>>>>>>>>>> 1)
>>>>>>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2)
>>>>>>>>>> public void feed(final Buffer buffer, final boolean last)
>>>>>>>>>> throws IOException {
>>>>>>>>>> try {
>>>>>>>>>> queue.put(new BodyPart(buffer, last));
>>>>>>>>>> } catch (InterruptedException e) {
>>>>>>>>>> throw new RuntimeException(e);
>>>>>>>>>> }
>>>>>>>>>> queueSize.incrementAndGet();
>>>>>>>>>>
>>>>>>>>>> if (context != null) {
>>>>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>>>>> flushQueue(true);
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> private void
>>>>>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>> try { Thread.sleep(10); } catch ( Exception e ) { throw
>>>>>>>>>> new RuntimeException(e); }
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> private boolean connectionIsReadyToWrite(FilterChainContext
>>>>>>>>>> fcc) {
>>>>>>>>>> Connection connection = fcc.getConnection();
>>>>>>>>>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>>>>>>>>>> return sslEngine != null &&
>>>>>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We had come to similar conclusions on this end. I'm still
>>>>>>>>>> working through testing the idea I mentioned previously (took longer than I
>>>>>>>>>> expected - sorry).
>>>>>>>>>> I hope to have something for you to test very soon.
>>>>>>>>>>
>>>>>>>>>> Note that it will be taking the above into account as well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have tried to put a while ( context == null ) Thread.sleep but
>>>>>>>>>>> it doesn't seem to work, when the context gets injected, after the sleeps,
>>>>>>>>>>> there's an OOM
>>>>>>>>>>>
>>>>>>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have done another test, remember my code that worked, which
>>>>>>>>>>> previously "warmed" the Thread with an useless request.
>>>>>>>>>>>
>>>>>>>>>>> private Runnable uploadSingleDocumentRunnable = new Runnable()
>>>>>>>>>>> {
>>>>>>>>>>> @Override
>>>>>>>>>>> public void run() {
>>>>>>>>>>> try {
>>>>>>>>>>> getUselessSessionCode();
>>>>>>>>>>> Thread.sleep(X);
>>>>>>>>>>> uploadSingleDocument();
>>>>>>>>>>> } catch ( Exception e ) {
>>>>>>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>> };
>>>>>>>>>>>
>>>>>>>>>>> I have put a sleep of X between the useless warmup request, and
>>>>>>>>>>> the real upload request
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What I noticed is that there is a very different behavior
>>>>>>>>>>> according to the value of X
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Under 10 seconds, it seems the stuff is still warm, I can upload
>>>>>>>>>>> the documents.
>>>>>>>>>>> Around 10 seconds I get a stack which seems to be "connection
>>>>>>>>>>> closed" or something
>>>>>>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The stacks I get for 10 seconds looks like
>>>>>>>>>>>
>>>>>>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>
>>>>>>>>>>> I think I got some other different stacks saying Connection
>>>>>>>>>>> Closed Remotely or something like that.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So it seems that something is bound to my thread, and it stays
>>>>>>>>>>> bound to it for about 10 seconds, do you have any idea what it could be?
>>>>>>>>>>> (My connection timeout setting seems to have no effect on this
>>>>>>>>>>> 10s threshold)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>> That is one solution. I'm working out an alternative right
>>>>>>>>>>>> now. Stay tuned!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway it's not a problem, I think the
>>>>>>>>>>>> FeedableBodyGenerator.feed() method just has to block until a context has
>>>>>>>>>>>> been (and ThreadCache initialized) to avoid OOM errors
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is very strange is that I tested with/without the same
>>>>>>>>>>>>> sessionCode with our previous code, the one not using
>>>>>>>>>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>>>>>>>>>> Despites the fact it had high memory consumption, it seems
>>>>>>>>>>>>> work fine to upload multiple documents if allocated with a large heap, and
>>>>>>>>>>>>> the sessionCode seems to have no effect.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On the new impl using the FeedableBodyGenerator, the
>>>>>>>>>>>>> sessionCode sent as a multipart bodypart seems to have an effect.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried to feed the queue before sending the request to
>>>>>>>>>>>>> AHC, but this leads to this exception (with/without sessionCode switching)
>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout
>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> By the way, by using a low timeout with the same sessioncode,
>>>>>>>>>>>>>> I got the following NPE:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>>>>>>>> httpCtx =
>>>>>>>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is right, here's a log I have when I use the same
>>>>>>>>>>>>>>> session code, ie the remote host is blocking the data or something.
>>>>>>>>>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>>>>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>>>>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>>>>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268',
>>>>>>>>>>>>>>> transport: 'socket'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Otherwise, with different session codes, I get the following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>> ... and this continues without OOM
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, this seems to be the problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it would be great to be able to be able to choose
>>>>>>>>>>>>>>> the queue impl behind that FeedableBodyGenerator, like I suggested in my
>>>>>>>>>>>>>>> pull request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> See here:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in this
>>>>>>>>>>>>>>> context, and in my case I would probably use a queue of size 1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This would handle the blocking of the feed method, without
>>>>>>>>>>>>>>> having to use this:
>>>>>>>>>>>>>>> if (context != null) {
>>>>>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Or perhaps the feed() method have to wait until a context is
>>>>>>>>>>>>>>> set in the BodyGenerator ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it would be more clear if
>>>>>>>>>>>>>>> the initializeAsynchronousTransfer simply didn't flush the queue but just
>>>>>>>>>>>>>>> setup the context.
>>>>>>>>>>>>>>> Then the feed method would block until there's a context
>>>>>>>>>>>>>>> set, and then flush the queue with blocking behavior.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is probably the next step, but as we are using AHC for
>>>>>>>>>>>>>>> async, it would probably be great if that blocking feed() method was called
>>>>>>>>>>>>>>> in a worker thread instead of our main thread.
>>>>>>>>>>>>>>> I won't use this but someone who really wants a non-blocking
>>>>>>>>>>>>>>> impl of performant multipart fileupload would probably need this, or will
>>>>>>>>>>>>>>> use an ExecutorService for the feeding operations as a workaround.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks again for your reactivity
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've integrated this change and it works fine except a
>>>>>>>>>>>>>>>> little detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>>>>>>>>>> This API requires a "sessionCode" in each request. So there
>>>>>>>>>>>>>>>> is a multipart StringPart with that SessionCode.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We used to have a cache which holds the sessionCode 30min
>>>>>>>>>>>>>>>> per user so that we do not need to init a new session each time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I had troubles in this very specific case: when I upload 5
>>>>>>>>>>>>>>>> docs with the same session code.
>>>>>>>>>>>>>>>> When I remove the cache and use 5 different session codes,
>>>>>>>>>>>>>>>> it works fine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I guess the remote service is blocking concurrent uploads
>>>>>>>>>>>>>>>> with the same session code. I don't know at all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Where I want to go is that I wouldn't have expected Grizzly
>>>>>>>>>>>>>>>> to OOM
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of
>>>>>>>>>>>>>>>> each Buffer sent to the FeedableBodyGenerator
>>>>>>>>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you have any idea why is there an OOM with these
>>>>>>>>>>>>>>>> settings?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Perhaps it is because the feed() method of
>>>>>>>>>>>>>>>> FeedableBodyGenerator doesn't block until the context is initialized.
>>>>>>>>>>>>>>>> I guess the initializeAsynchronousTransfer is called only
>>>>>>>>>>>>>>>> once the connection is established, and perhaps a lot of Buffer are added
>>>>>>>>>>>>>>>> to the queue...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, it's invoked once the request has been dispatched, so
>>>>>>>>>>>>>>>> if the generator is fed a lot before the request, I could see this
>>>>>>>>>>>>>>>> happening.
>>>>>>>>>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But I'm not sure at all because the session code is
>>>>>>>>>>>>>>>> transmitted as a BodyPart and I get the same problem if i put it as the
>>>>>>>>>>>>>>>> first or last multipart.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's not a big deal, perhaps I should always use a
>>>>>>>>>>>>>>>> different session code for concurrent operations but I'd like to be sure
>>>>>>>>>>>>>>>> that we won't have this issue in production...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Good catch. Fixed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There's a little mistake in the grizzly ahc provider
>>>>>>>>>>>>>>>>> relative to the write queue size.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As you can see, the TransportCustomizer is called, and
>>>>>>>>>>>>>>>>> then the write queue size (among other things) is set to AUTO_SIZE (instead
>>>>>>>>>>>>>>>>> of previously UNLIMITED)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think the default settings like this AUTO_SIZE attribute
>>>>>>>>>>>>>>>>> should be set before the customization of the transport, or they would
>>>>>>>>>>>>>>>>> override the value we customized.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is actually my case, since I can't reproduce my "bug"
>>>>>>>>>>>>>>>>> which is "high memory consumption", even when using -1 / UNLIMITED in the
>>>>>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd rather
>>>>>>>>>>>>>>>>> be able to tune this parameter during load tests to see the effect.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a
>>>>>>>>>>>>>>>>>> 500m file with 40m heap size ;)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <
>>>>>>>>>>>>>>>>>> ryan.lubke_at_oracle.com> a écrit :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm going to be updating the Grizzly provider such that
>>>>>>>>>>>>>>>>>> AUTO_SIZE (not AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding your tuning question, I would probably set the
>>>>>>>>>>>>>>>>>> value to AsyncQueueWriter.AUTO_TUNE (this will be four times the socket
>>>>>>>>>>>>>>>>>> write buffer) and see how that works.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> A question first. With these changes, your memory usage
>>>>>>>>>>>>>>>>>> is more inline with what you were looking for?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will be
>>>>>>>>>>>>>>>>>> released (with these new improvements?)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We would like to know if we wait for a release or if we
>>>>>>>>>>>>>>>>>> install our own temp release on Nexus :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If I let the initial value (-1) it won't block, canWrite
>>>>>>>>>>>>>>>>>>> always returns true
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value as
>>>>>>>>>>>>>>>>>>> a config attribute, neither the size of the write buffer you talked about.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So in the end, is there a way with current AHC code to
>>>>>>>>>>>>>>>>>>> use this "canWrite = false" behavior?
>>>>>>>>>>>>>>>>>>> If not, can you please provide a way to set this
>>>>>>>>>>>>>>>>>>> behavior on v1.7.20 ? thanks
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> PS: does it make sens to use the same number of bytes un
>>>>>>>>>>>>>>>>>>> the feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Could you also provide a sample of how you're
>>>>>>>>>>>>>>>>>>>> performing your feed?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> By default, Grizzly will configure the async write
>>>>>>>>>>>>>>>>>>>> queue length to be four times the write buffer size (which is based off the
>>>>>>>>>>>>>>>>>>>> socket write buffer).
>>>>>>>>>>>>>>>>>>>> When this queue exceeds this value, canWrite() will
>>>>>>>>>>>>>>>>>>>> return false.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> When this occurs, you can register a WriteListener to
>>>>>>>>>>>>>>>>>>>> be notified when the queue length is below the configured max and then
>>>>>>>>>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>>>>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>> public void onWritePossible() throws
>>>>>>>>>>>>>>>>>>>> Exception {
>>>>>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>>>>>>>>>> I use AHC default which seems to
>>>>>>>>>>>>>>>>>>>> use SameThreadIOStrategy so I don't think it's related to the IO strategy.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of
>>>>>>>>>>>>>>>>>>>> 50m, but I have to put a Thread.sleep(30) between each 100k Buffer send to
>>>>>>>>>>>>>>>>>>>> the FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The connection with the server is not good here, but in
>>>>>>>>>>>>>>>>>>>> production it is normally a lot better as far as I know.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've tried things
>>>>>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'd like the Grizzly internals to block when there are
>>>>>>>>>>>>>>>>>>>> too much pending bytes to send. Is it possible?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> PS: I've just been able to send a 500mo file with 100mo
>>>>>>>>>>>>>>>>>>>> heap, but it needed a sleep of 100ms between each 100k Buffer sent to the
>>>>>>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner
>>>>>>>>>>>>>>>>>>>>> used in the SSL filter?
>>>>>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>>>>>>>>>> I don't really understand why messages have to be
>>>>>>>>>>>>>>>>>>>>> cloned, can I remove this? How?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a
>>>>>>>>>>>>>>>>>>>>>> heap of 400m.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In our real use cases we would probably have files
>>>>>>>>>>>>>>>>>>>>>> under 20mo but we want to reduce the memory consumption because we can have
>>>>>>>>>>>>>>>>>>>>>> x parallel uploads on the same server according to the user activity.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator reduced
>>>>>>>>>>>>>>>>>>>>>> the memory footprint or if it's almost like before.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer
>>>>>>>>>>>>>>>>>>>>>>> allocation is concerned, it's untunable.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As to your second question, there is no suggested
>>>>>>>>>>>>>>>>>>>>>>> size. This is all very application specific.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of
>>>>>>>>>>>>>>>>>>>>>>> 33842 and it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so
>>>>>>>>>>>>>>>>>>>>>>> that it consumes less memory?
>>>>>>>>>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed
>>>>>>>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each new
>>>>>>>>>>>>>>>>>>>>>>>> buffer that is passed to the SSLFilter, we allocate a buffer twice the size
>>>>>>>>>>>>>>>>>>>>>>>> in order to
>>>>>>>>>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an
>>>>>>>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Depending on the socket configurations of both
>>>>>>>>>>>>>>>>>>>>>>>> endpoints, and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>>>>>>>>>> be the write queue is becoming too large. We do
>>>>>>>>>>>>>>>>>>>>>>>> have a way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded here.
>>>>>>>>>>>>>>>>>>>>>>>> I will see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded like
>>>>>>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream the
>>>>>>>>>>>>>>>>>>>>>>>> body or something like that?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull
>>>>>>>>>>>>>>>>>>>>>>>>> request on AHC tomorrow with a better code using the same Part classes that
>>>>>>>>>>>>>>>>>>>>>>>>> already exist.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder
>>>>>>>>>>>>>>>>>>>>>>>>> feeds the queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or
>>>>>>>>>>>>>>>>>>>>>>>>> something, will work on that, it will be better than a thread sleep to slow
>>>>>>>>>>>>>>>>>>>>>>>>> down the filepart reading
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to tackle
>>>>>>>>>>>>>>>>>>>>>>>>>> adding something like this today. I'll follow up with something you can
>>>>>>>>>>>>>>>>>>>>>>>>>> test out.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think I see what I could do, probably something
>>>>>>>>>>>>>>>>>>>>>>>>>> like that:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder = new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new
>>>>>>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes =
>>>>>>>>>>>>>>>>>>>>>>>>>> asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to upload
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a file with AHC / Grizzly in streaming, I mean without loading the whole
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file bytes in memory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a byte[]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> which contans the whole file, so it means that my server can be OOM if too
>>>>>>>>>>>>>>>>>>>>>>>>>>>> many users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory
>>>>>>>>>>>>>>>>>>>>>>>>>>>> managers, with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended wto
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC
>>>>>>>>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file so that I could eventually consume only 20mo of heap while uploading a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 500mo file?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>> But if you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area can
>>>>>>>>>>>>>>>>>>>>>>>>>>> be improved.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file with CXF and have to transmit it to a storage server (like S3). CXF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't consume memory bevause it is streaming the large fle uploads to the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>