From: Sébastien Lorber <lorber.sebastien_at_gmail.com>
Date: Fri, 20 Sep 2013 18:02:08 +0200

Yes it works nicely :)

Thanks Ryan I think everything is now working for me


2013/9/20 Ryan Lubke <ryan.lubke_at_oracle.com>

> Just following up for closure here. Pool issue has been confirmed as
> resolved.
>
> Sébastien Lorber wrote:
>
> Ok thanks :)
>
> Take your time
> We'll keep using no connection pool (as we used to do until now anyway,
> and we could probably continue with this setting for a while because it
> passed our load test goals)
>
>
> 2013/9/19 Ryan Lubke <ryan.lubke_at_oracle.com>
>
>> Sorry, been under the weather. I'll be investigating this issue this
>> morning.
>>
>> Stay tuned.
>>
>> Sébastien Lorber wrote:
>>
>> So what I've found is that:
>>
>>
>> private final Connection.CloseListener listener = new
>> Connection.CloseListener() {
>> @Override
>> public void onClosed(Connection connection,
>> Connection.CloseType closeType) throws IOException {
>> if (closeType == Connection.CloseType.REMOTELY) {
>> if (LOG.isInfoEnabled()) {
>> LOG.info("Remote closed connection ({}).
>> Removing from cache", connection.toString());
>> }
>> }
>> GrizzlyConnectionsPool.this.removeAll(connection);
>> }
>> };
>>
>>
>> public boolean removeAll(Connection connection) {
>>
>> if (connection == null || closed.get()) {
>> return false;
>> }
>> connection.removeCloseListener(listener);
>> boolean isRemoved = false;
>> for (Map.Entry<String, DelayedExecutor.IdleConnectionQueue> entry
>> : connectionsPool.entrySet()) {
>> boolean removed = entry.getValue().remove(connection);
>> isRemoved |= removed;
>> }
>> return isRemoved;
>>
>> }
>>
>>
>>
>> When the connection is closed remotely, the connection that we try to
>> remove is never in the cache, thus isRemoved = false
>>
>> I guess it has something to do with the "connection discrimination" you
>> previously fixed and I didn't really understand :)
>>
>> Do you have an idea on this problem?
>>
>>
>>
>>
>>
>> By the way, I just tested again WITHOUT the FeedableBodyGenerator.
>> I run batches of 10 concurrent uploads with a max 20 connection pool.
>> I run these 10 concurrent uploads multiple times.
>> It works fine, but if just after the request I add a Thread.sleep(15000)
>> then it seems to lead to the same problem:
>>
>> @Test
>> public void do_test_without_ahc_streaming() throws Exception {
>> List<Response> responses = runConcurrently(CONCURRENT_UPLOADS,new
>> Callable<Response>() {
>> @Override
>> public Response call() throws Exception {
>> return uploadSingleDocumentWithoutStreaming();
>> }
>> });
>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>> @Override
>> public Response call() throws Exception {
>> return uploadSingleDocumentWithoutStreaming();
>> }
>> });
>> Thread.sleep(15000);
>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>> @Override
>> public Response call() throws Exception {
>> return uploadSingleDocumentWithoutStreaming();
>> }
>> });
>> Thread.sleep(15000);
>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>> @Override
>> public Response call() throws Exception {
>> return uploadSingleDocumentWithoutStreaming();
>> }
>> });
>> Thread.sleep(15000);
>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>> @Override
>> public Response call() throws Exception {
>> return uploadSingleDocumentWithoutStreaming();
>> }
>> });
>> Thread.sleep(15000);
>> ......................................;
>> }
>>
>> This permits to give some time for the remote host to close the connection
>>
>>
>>
>> So in the end it seems that we have troubles in any case when the remote
>> host closes the connection
>>
>>
>>
>>
>> Btw I just saw this issue:
>> https://github.com/AsyncHttpClient/async-http-client/pull/311
>> I checked the code that used AHC 1.7.7 and what I can see is that the
>> connection pool is bypassed even with the property=true
>> I have in my logs:
>> [main] DEBUG
>> com.ning.http.client.providers.grizzly.GrizzlyConnectionsPool:162 - [poll]
>> No existing queue for uri [https://digiposte.orsid.com:443].
>>
>>
>> So, by updating to 1.7.20-SNAPSHOT, the connection pool has been
>> magically enabled, and it doesn't seem to work well for me.
>>
>>
>> I'll keep that pool disabled because it's the behavior of our previous
>> applicative version that is near to production and works pretty fine, but
>> will open a Github issue because i'm quitting my job next month and they
>> want to track when they'll be able to use a connection pool :)
>>
>>
>>
>> 2013/9/19 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>
>>> So it seems I can reproduce this in local.
>>>
>>> Disabling the ssl connection pooling solve the problem.
>>>
>>>
>>> We didn't have this problems with v1.7.7 we previously used.
>>>
>>> I just tested with the last commits of 1.7.20-SNAPSHOT, the problem is
>>> still here.
>>> The problem doesn't appear on normal AHC requests, neither on multipart
>>> file upload when I do not use the FeedableBodyGenerator so it seems this is
>>> the new behavior of FeedableBodyGenerator that has a problem with the ssl
>>> connection pooling. Will try to investigate this.
>>>
>>> By the way, your last commits about selector/worker threads seems ok
>>> (didn't notice any problem with the ssl pool disabled)
>>>
>>>
>>>
>>> 2013/9/18 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>
>>>> Hi
>>>>
>>>> Unfortunatly it seems there's another problem.
>>>> On our test environment I often get in the logs:
>>>>
>>>> Caught internal server errorjava.io.IOException: Maximum pooled
>>>> connections exceeded
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.cleanup(GrizzlyAsyncHttpProvider.java:1415)
>>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpPacketParsed(GrizzlyAsyncHttpProvider.java:1366)
>>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>>> at
>>>> org.glassfish.grizzly.http.HttpCodecFilter.decodeWithTransferEncoding(HttpCodecFilter.java:1176)
>>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:516)
>>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:161)
>>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> Wrapped by: java.util.concurrent.ExecutionException:
>>>> java.io.IOException: Maximum pooled connections exceeded
>>>> at
>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:359)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:265)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyResponseFuture.get(GrizzlyResponseFuture.java:165)
>>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>>>
>>>>
>>>>
>>>>
>>>> boolean canReturnConnection(final Connection c) {
>>>>
>>>> return (DO_NOT_CACHE.get(c) != null ||
>>>> pool.canCacheConnection());
>>>>
>>>> }
>>>>
>>>>
>>>> private static HttpTransactionContext cleanup(final
>>>> FilterChainContext ctx,
>>>> final
>>>> GrizzlyAsyncHttpProvider provider) {
>>>>
>>>> final Connection c = ctx.getConnection();
>>>> final HttpTransactionContext context =
>>>> provider.getHttpTransactionContext(c);
>>>> context.provider.setHttpTransactionContext(c, null);
>>>> if
>>>> (!context.provider.connectionManager.canReturnConnection(c)) {
>>>> context.abort(new IOException("Maximum pooled
>>>> connections exceeded"));
>>>> } else {
>>>> if
>>>> (!context.provider.connectionManager.returnConnection(context.request, c)) {
>>>> ctx.getConnection().close();
>>>> }
>>>> }
>>>>
>>>> return context;
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> We use:
>>>>
>>>> Key=[properties.httpclient.allowSslConnectionPool] Value=[true]
>>>> Key=[properties.httpclient.maximumConnectionsPerHost] Value=[20]
>>>> Key=[properties.httpclient.maximumConnectionsTotal] Value=[30]
>>>> Key=[properties.httpclient.timeout.connection] Value=[10000]
>>>> Key=[properties.httpclient.timeout.request] Value=[30000]
>>>>
>>>>
>>>> In our logs, I can see there are 30 times this log:
>>>>
>>>> Remote closed connection (TCPNIOConnection{localSocketAddress={/
>>>> 172.16.104.160:55488}, peerSocketAddress={host/172.16.4.100:443}}).
>>>> Removing from cache
>>>>
>>>> and then it seems no connection is never available and the app is
>>>> blocked.
>>>>
>>>> I'll try tomorrow to disable or increase the pool size, but if you have
>>>> any idea of the problem please let me know.
>>>> We do not run load tests, these are simple functional tests with nearly
>>>> no concurrency.
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/17 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>
>>>>> This is good to hear. Thanks for all the feedback and working with us
>>>>> to nail this down.
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>
>>>>> Thanks,
>>>>>
>>>>> We already installed the previous snapshot in our nexus because it
>>>>> works fine and I'm working on something else now but I'll give you a
>>>>> feedback soon to see if this still works fine.
>>>>> For us it doesn't really mater which thread is using since we do not
>>>>> use Future in our applications for this case.
>>>>>
>>>>>
>>>>> Btw I've been able deploy a main with my code running concurrent
>>>>> uploads on a server which has a better connectivity with the remote API and
>>>>> it seems I can upload up to 150 concurrent files of 10mb with a heap of
>>>>> 250mo in 30 seconds and a throughput near 50mb/s
>>>>>
>>>>> I don't really know the infrastructure on which my code was deployed
>>>>> but I suspect Grizzly is not the bottleneck anymore :)
>>>>> When I have more than 150 concurrent uploads the remote endpoint seems
>>>>> to timeout under the load.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/16 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>
>>>>>>
>>>>>>
>>>>>> Ryan Lubke wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I noticed something strange.
>>>>>> On the FileInputStream I have, I've added a log on the close() of the
>>>>>> stream which is called once the whole file has been read to be sent to the
>>>>>> feeder.
>>>>>>
>>>>>> 1) If I perform a request (ie my session init request in the previous
>>>>>> discussions) before doing my multipart upload, the thread that does execute
>>>>>> the feeding is the thread that fires the request, and not a Grizzly worker.
>>>>>> [Thread-30] INFO Will close file stream
>>>>>> java.io.FileInputStream_at_27fe4315 after having read 1440304 bytes
>>>>>>
>>>>>>
>>>>>> 2) If I don't do any request before firing the multipart upload, the
>>>>>> thread that does the feeding is a Grizzly threadpool worker thread:
>>>>>> [Grizzly(22)] INFO Will close file stream
>>>>>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>>>>>
>>>>>>
>>>>>> Is this normal? I would expect a worker thread to always be used, and
>>>>>> the main applicative thread that performs the request to never be blocking.
>>>>>> (but it's not such an issue for me, we don't have a reactive non-blocking
>>>>>> app anyway)
>>>>>>
>>>>>> It's currently expected behavior. Will need to re-evaluate this
>>>>>> based on the new semantics of the FeedableBodyGenerator.
>>>>>>
>>>>>> I've committed a change for this. If, upon testing, you find there
>>>>>> is still an issue, please let us know.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On the 1st case, here's the stacktrace when the feed method is called:
>>>>>>
>>>>>> * at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)
>>>>>> *
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
>>>>>> at
>>>>>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>>>>>> at
>>>>>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
>>>>>> * at
>>>>>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)
>>>>>> *
>>>>>>
>>>>>> So it seems this happen when the handshake has already been done when
>>>>>> *initializeAsynchronousTransfer* is called, so that we do not go
>>>>>> through the HandshakeListener
>>>>>>
>>>>>>
>>>>>> Notice that this case seems to also happen in a few threads on the
>>>>>> 2nd case, but most of the threads are Grizzly workers.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>>
>>>>>>> Thanks, it seems to work.
>>>>>>>
>>>>>>> I would suggest to throw IOException on the flush method since the
>>>>>>> feed method is supposed to be called here
>>>>>>>
>>>>>>>
>>>>>>> My implementation of flush is:
>>>>>>>
>>>>>>> @Override
>>>>>>> public void flush() {
>>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>>> try ( OutputStream outputStream = createFeedingOutputStream()
>>>>>>> ) {
>>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>>> } catch (Exception e) {
>>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>>> FeedableBodyGenerator",e);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Is this correct? The OutputStream redirects the bytes written to the
>>>>>>> feed(Buffer) method
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> There seems to be some concurrency issue. Because the upload of 1
>>>>>>> file seems fine, but when using multiple threads, I often get the following
>>>>>>> stack:
>>>>>>> Caused by: java.io.IOException: Stream Closed
>>>>>>> at java.io.FileInputStream.readBytes(Native Method)
>>>>>>> at java.io.FileInputStream.read(FileInputStream.java:242)
>>>>>>> at
>>>>>>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
>>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>
>>>>>>>
>>>>>>> This is because the flush() method is called multiple times for the
>>>>>>> same request on some cases.
>>>>>>> I guess this is not supposed to happen.
>>>>>>> What I understand is that the flush() method is supposed to be
>>>>>>> called only once.
>>>>>>>
>>>>>>>
>>>>>>> Using debug logging breakpoints I get the following:
>>>>>>>
>>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19 started
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>>> *Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> *
>>>>>>> *Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> *
>>>>>>> *Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> *
>>>>>>> *Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> *
>>>>>>> *Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> *
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>>>>>>
>>>>>>>
>>>>>>> As you can see the same HandshakeListener seems to be called
>>>>>>> multiple times for the same FeedableBodyGenerator
>>>>>>>
>>>>>>> When this happens, the stack is:
>>>>>>>
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So I tried with the following code:
>>>>>>>
>>>>>>> private boolean alreadyFlushed = false;
>>>>>>> @Override
>>>>>>> public synchronized void flush() {
>>>>>>> if ( alreadyFlushed ) {
>>>>>>> return;
>>>>>>> }
>>>>>>> startFeeding();
>>>>>>> alreadyFlushed = true;
>>>>>>> }
>>>>>>>
>>>>>>> It works fine when upload small files.
>>>>>>>
>>>>>>> But with larger files, I often get TimeoutException stacks for some
>>>>>>> of the threads:
>>>>>>>
>>>>>>> Caused by: java.util.concurrent.TimeoutException
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Did I miss something?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>> Committed another small change. Please make sure you're at the
>>>>>>>> latest when you build.
>>>>>>>>
>>>>>>>> -rl
>>>>>>>>
>>>>>>>>
>>>>>>>> Ryan Lubke wrote:
>>>>>>>>
>>>>>>>> Okay, I've committed another set of refactorings to the
>>>>>>>> FeedableBodyGenerator.
>>>>>>>>
>>>>>>>> For your use case, you should extend
>>>>>>>> FeedableBodyGenerator.SimpleFeeder.
>>>>>>>>
>>>>>>>> Let me know if you run into issues.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>
>>>>>>>> Yes I think it would, so that I can feed the queue at once
>>>>>>>>
>>>>>>>> One thread will be locked during the feeding for nothing but it's
>>>>>>>> not a real problem in my usecase.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>
>>>>>>>>> Would having a different listener that will be notified once async
>>>>>>>>> transferring has been started work better for you?
>>>>>>>>>
>>>>>>>>> Something like:
>>>>>>>>>
>>>>>>>>> onAsyncTransferInitiated() {
>>>>>>>>> // invoke your feed method
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>
>>>>>>>>> Unfortunatly I won't be able to use the Feeder non-blocking stuff
>>>>>>>>> for now, because of how the multipart request in handled in AHC
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here's my feeding method:
>>>>>>>>>
>>>>>>>>> public void feed() throws IOException {
>>>>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>>>>> try ( OutputStream outputStream = createFeedingOutputStream()
>>>>>>>>> ) {
>>>>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>>>>> } catch (Exception e) {
>>>>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>>>>> FeedableBodyGenerator",e);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As you can see, the multipart Parts array can only be pushed to
>>>>>>>>> the OutputStream, I don't have any way to "pull" the data when the
>>>>>>>>> canFeed() method is triggered.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But I've seen that there's
>>>>>>>>> a com.ning.http.multipart.MultipartBody#read that seems to provide a memory
>>>>>>>>> efficient way to pull data from a Multipart body...
>>>>>>>>>
>>>>>>>>> Should see what I come up with this
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>
>>>>>>>>>> It seems the Feeder is highly recommended but not mandatory so I
>>>>>>>>>> tried without.
>>>>>>>>>>
>>>>>>>>>> With my existing code it seems there is a synchronization problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The feeding threads get locked to the prematureFeed.get();
>>>>>>>>>>
>>>>>>>>>> So the Grizzly kernel threads are unable to acquire the lock
>>>>>>>>>> required to enter the initializeAsynchronousTransfer method
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Will try with an implementation of Feeder
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hmmm it seems I have a problem with one of your maven plugins.
>>>>>>>>>>> I'll try to bypass it, but for info:
>>>>>>>>>>>
>>>>>>>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>>>>>>>> [WARNING]
>>>>>>>>>>> [WARNING] Some problems were encountered while building the
>>>>>>>>>>> effective settings
>>>>>>>>>>> [WARNING] 'profiles.profile[default].repositories.repository.id'
>>>>>>>>>>> must be unique but found duplicate repository with id
>>>>>>>>>>> fullsix-maven-repository @ /home/slorber/.m2/settings.xml
>>>>>>>>>>> [WARNING]
>>>>>>>>>>> [INFO] Scanning for projects...
>>>>>>>>>>> [INFO]
>>>>>>>>>>>
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>>>>>>>>>> async-http-client ---
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>>>>>>>>>>> (enforce-maven) @ async-http-client ---
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>>>>>>>>>>> (enforce-versions) @ async-http-client ---
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] --- maven-resources-plugin:2.4.3:resources
>>>>>>>>>>> (default-resources) @ async-http-client ---
>>>>>>>>>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>>>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>>>>>>>> [INFO]
>>>>>>>>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile)
>>>>>>>>>>> @ async-http-client ---
>>>>>>>>>>> [INFO] Compiling 158 source files to
>>>>>>>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>>>>>>>> [INFO]
>>>>>>>>>>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>>>>>>>>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>>>>>>>>> *[INFO] Checking unresolved references to
>>>>>>>>>>> org.codehaus.mojo.signature:java15:1.0*
>>>>>>>>>>> *[ERROR] Undefined reference:
>>>>>>>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>>>>>>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
>>>>>>>>>>> *
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> [INFO] Total time: 8.747s
>>>>>>>>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>>>>>>>>> [INFO] Final Memory: 30M/453M
>>>>>>>>>>> [INFO]
>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>> *[ERROR] Failed to execute goal
>>>>>>>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>>>>>>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>>>>>>>>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]
>>>>>>>>>>> *
>>>>>>>>>>> [ERROR]
>>>>>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
>>>>>>>>>>> with the -e switch.
>>>>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
>>>>>>>>>>> logging.
>>>>>>>>>>> [ERROR]
>>>>>>>>>>> [ERROR] For more information about the errors and possible
>>>>>>>>>>> solutions, please read the following articles:
>>>>>>>>>>> [ERROR] [Help 1]
>>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Ok thank you, I'll try to implement that today and will give
>>>>>>>>>>>> you my feedback :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Okay,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've committed my initial changes to the AHC repository.
>>>>>>>>>>>>> Here's a summary of the changes:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>>>>>>>>>> - Don't allow queueing of data before initiateAsyncTransfer
>>>>>>>>>>>>> has been invoked. In low memory
>>>>>>>>>>>>> heaps, this could lead to an OOM if the source is feeding too
>>>>>>>>>>>>> fast. The new behavior is to
>>>>>>>>>>>>> block until initiateAsyncTransfer is called, at which time the
>>>>>>>>>>>>> blocked thread may proceed with
>>>>>>>>>>>>> the feed operation.
>>>>>>>>>>>>> - Introduce the concept of a Feeder. Implementations are
>>>>>>>>>>>>> responsible, at a high level, for:
>>>>>>>>>>>>> + letting the provider know that data is available to be fed
>>>>>>>>>>>>> without blocking
>>>>>>>>>>>>> + allowing the registration of a callback that the Feeder
>>>>>>>>>>>>> implementation may invoke
>>>>>>>>>>>>> to signal that more data is available, if it wasn't available
>>>>>>>>>>>>> at a previous point in time.
>>>>>>>>>>>>> - When using a Feeder with a secure request, the SSL handshake
>>>>>>>>>>>>> will be kicked off by the
>>>>>>>>>>>>> initiateAsyncTransfer call, but feeding of data will not occur
>>>>>>>>>>>>> until the handshake is complete.
>>>>>>>>>>>>> This is necessary as the SSLFilter will queue up all writes
>>>>>>>>>>>>> until the handshake is complete,
>>>>>>>>>>>>> and currently, the buffer isn't tied in with the transport
>>>>>>>>>>>>> flow control mechanism.
>>>>>>>>>>>>> NOTE: This new SSL behavior is not currently applied when
>>>>>>>>>>>>> invoking the feed() method
>>>>>>>>>>>>> outside the context of a Feeder. Still need to address that.
>>>>>>>>>>>>> - Exposed configuration of the async write queue limit through
>>>>>>>>>>>>> the FeedableBodyGenerator.
>>>>>>>>>>>>> This is an improvement on using a TransportCustomizer as any
>>>>>>>>>>>>> configuration there is
>>>>>>>>>>>>> transport-wide, and therefor applied to all Connections. By
>>>>>>>>>>>>> exposing it here, each feeder
>>>>>>>>>>>>> may have a different byte limit.
>>>>>>>>>>>>> - Improved documentation for this class*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I recommend reading through the javadoc comments in the source
>>>>>>>>>>>>> [1] for FeedableBodyGenerator (comments welcome).
>>>>>>>>>>>>> Additionally, I would re-work your code to leverage the Feeder
>>>>>>>>>>>>> instead of calling feed() directly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you have issues implementing Feeder, do let us know.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you have additional questions, again, let us know.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> So in the end I've end up with an implementation that's
>>>>>>>>>>>>> working for me.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think there are 2 bugs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue
>>>>>>>>>>>>> if the initialize(ctx) method is not called fast enough.
>>>>>>>>>>>>> This can be solved by using a BlockingQueue of size 1 and the
>>>>>>>>>>>>> put() method.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Once the context is injected, the FeedableBodyGenerator
>>>>>>>>>>>>> flushes the queue.
>>>>>>>>>>>>> The matter is that if the connection is new, not warmed up by
>>>>>>>>>>>>> a previous request, then the SSL handshake is not done yet, and it seems
>>>>>>>>>>>>> that the bytes are accumulated in some part of the SSL filter which doesn't
>>>>>>>>>>>>> deliver them to the connection until the handshake has completed,
>>>>>>>>>>>>> so c.canWrite() continues to return true.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have replaced some part of the FeedableBodyGenerator to test
>>>>>>>>>>>>> this and it works pretty fine. See what I have changed:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1)
>>>>>>>>>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>>>>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2)
>>>>>>>>>>>>> public void feed(final Buffer buffer, final boolean last)
>>>>>>>>>>>>> throws IOException {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> queue.put(new BodyPart(buffer, last));
>>>>>>>>>>>>> } catch (InterruptedException e) {
>>>>>>>>>>>>> throw new RuntimeException(e);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> queueSize.incrementAndGet();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (context != null) {
>>>>>>>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void
>>>>>>>>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>>>>>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>>>>> try { Thread.sleep(10); } catch ( Exception e ) {
>>>>>>>>>>>>> throw new RuntimeException(e); }
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> private boolean
>>>>>>>>>>>>> connectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>>>>>>>>> Connection connection = fcc.getConnection();
>>>>>>>>>>>>> SSLEngine sslEngine =
>>>>>>>>>>>>> SSLUtils.getSSLEngine(connection);
>>>>>>>>>>>>> return sslEngine != null &&
>>>>>>>>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We had come to similar conclusions on this end. I'm still
>>>>>>>>>>>>> working through testing the idea I mentioned previously (took longer than I
>>>>>>>>>>>>> expected - sorry).
>>>>>>>>>>>>> I hope to have something for you to test very soon.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that it will be taking the above into account as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tried to put a while ( context == null ) Thread.sleep
>>>>>>>>>>>>>> but it doesn't seem to work, when the context gets injected, after the
>>>>>>>>>>>>>> sleeps, there's an OOM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have done another test, remember my code that worked, which
>>>>>>>>>>>>>> previously "warmed" the Thread with an useless request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private Runnable uploadSingleDocumentRunnable = new
>>>>>>>>>>>>>> Runnable() {
>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>> public void run() {
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>> getUselessSessionCode();
>>>>>>>>>>>>>> Thread.sleep(X);
>>>>>>>>>>>>>> uploadSingleDocument();
>>>>>>>>>>>>>> } catch ( Exception e ) {
>>>>>>>>>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have put a sleep of X between the useless warmup request,
>>>>>>>>>>>>>> and the real upload request
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What I noticed is that there is a very different behavior
>>>>>>>>>>>>>> according to the value of X
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Under 10 seconds, it seems the stuff is still warm, I can
>>>>>>>>>>>>>> upload the documents.
>>>>>>>>>>>>>> Around 10 seconds I get a stack which seems to be "connection
>>>>>>>>>>>>>> closed" or something
>>>>>>>>>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The stacks I get for 10 seconds looks like
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think I got some other different stacks saying Connection
>>>>>>>>>>>>>> Closed Remotely or something like that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So it seems that something is bound to my thread, and it
>>>>>>>>>>>>>> stays bound to it for about 10 seconds, do you have any idea what it could
>>>>>>>>>>>>>> be?
>>>>>>>>>>>>>> (My connection timeout setting seems to have no effect on
>>>>>>>>>>>>>> this 10s threshold)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That is one solution. I'm working out an alternative right
>>>>>>>>>>>>>>> now. Stay tuned!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyway it's not a problem, I think the
>>>>>>>>>>>>>>> FeedableBodyGenerator.feed() method just has to block until a context has
>>>>>>>>>>>>>>> been (and ThreadCache initialized) to avoid OOM errors
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What is very strange is that I tested with/without the same
>>>>>>>>>>>>>>>> sessionCode with our previous code, the one not using
>>>>>>>>>>>>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>>>>>>>>>>>>> Despites the fact it had high memory consumption, it seems
>>>>>>>>>>>>>>>> work fine to upload multiple documents if allocated with a large heap, and
>>>>>>>>>>>>>>>> the sessionCode seems to have no effect.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On the new impl using the FeedableBodyGenerator, the
>>>>>>>>>>>>>>>> sessionCode sent as a multipart bodypart seems to have an effect.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tried to feed the queue before sending the request
>>>>>>>>>>>>>>>> to AHC, but this leads to this exception (with/without sessionCode
>>>>>>>>>>>>>>>> switching)
>>>>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout
>>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By the way, by using a low timeout with the same
>>>>>>>>>>>>>>>>> sessioncode, I got the following NPE:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is right, here's a log I have when I use the same
>>>>>>>>>>>>>>>>>> session code, ie the remote host is blocking the data or something.
>>>>>>>>>>>>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>>>>>>>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Disconnected from the target VM, address: '
>>>>>>>>>>>>>>>>>> 127.0.0.1:49268', transport: 'socket'
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Otherwise, with different session codes, I get the
>>>>>>>>>>>>>>>>>> following:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>>> ... and this continues without OOM
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So, this seems to be the problem.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think it would be great to be able to be able to choose
>>>>>>>>>>>>>>>>>> the queue impl behind that FeedableBodyGenerator, like I suggested in my
>>>>>>>>>>>>>>>>>> pull request.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> See here:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in
>>>>>>>>>>>>>>>>>> this context, and in my case I would probably use a queue of size 1
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This would handle the blocking of the feed method,
>>>>>>>>>>>>>>>>>> without having to use this:
>>>>>>>>>>>>>>>>>> if (context != null) {
>>>>>>>>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Or perhaps the feed() method have to wait until a context
>>>>>>>>>>>>>>>>>> is set in the BodyGenerator ?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think it would be more clear if
>>>>>>>>>>>>>>>>>> the initializeAsynchronousTransfer simply didn't flush the queue but just
>>>>>>>>>>>>>>>>>> setup the context.
>>>>>>>>>>>>>>>>>> Then the feed method would block until there's a context
>>>>>>>>>>>>>>>>>> set, and then flush the queue with blocking behavior.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is probably the next step, but as we are using AHC
>>>>>>>>>>>>>>>>>> for async, it would probably be great if that blocking feed() method was
>>>>>>>>>>>>>>>>>> called in a worker thread instead of our main thread.
>>>>>>>>>>>>>>>>>> I won't use this but someone who really wants a
>>>>>>>>>>>>>>>>>> non-blocking impl of performant multipart fileupload would probably need
>>>>>>>>>>>>>>>>>> this, or will use an ExecutorService for the feeding operations as a
>>>>>>>>>>>>>>>>>> workaround.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks again for your reactivity
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've integrated this change and it works fine except a
>>>>>>>>>>>>>>>>>>> little detail.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>>>>>>>>>>>>> This API requires a "sessionCode" in each request. So
>>>>>>>>>>>>>>>>>>> there is a multipart StringPart with that SessionCode.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We used to have a cache which holds the sessionCode
>>>>>>>>>>>>>>>>>>> 30min per user so that we do not need to init a new session each time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had troubles in this very specific case: when I upload
>>>>>>>>>>>>>>>>>>> 5 docs with the same session code.
>>>>>>>>>>>>>>>>>>> When I remove the cache and use 5 different session
>>>>>>>>>>>>>>>>>>> codes, it works fine.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I guess the remote service is blocking concurrent
>>>>>>>>>>>>>>>>>>> uploads with the same session code. I don't know at all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Where I want to go is that I wouldn't have expected
>>>>>>>>>>>>>>>>>>> Grizzly to OOM
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of
>>>>>>>>>>>>>>>>>>> each Buffer sent to the FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>>>>>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Do you have any idea why is there an OOM with these
>>>>>>>>>>>>>>>>>>> settings?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Perhaps it is because the feed() method of
>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator doesn't block until the context is initialized.
>>>>>>>>>>>>>>>>>>> I guess the initializeAsynchronousTransfer is called
>>>>>>>>>>>>>>>>>>> only once the connection is established, and perhaps a lot of Buffer are
>>>>>>>>>>>>>>>>>>> added to the queue...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes, it's invoked once the request has been dispatched,
>>>>>>>>>>>>>>>>>>> so if the generator is fed a lot before the request, I could see this
>>>>>>>>>>>>>>>>>>> happening.
>>>>>>>>>>>>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But I'm not sure at all because the session code is
>>>>>>>>>>>>>>>>>>> transmitted as a BodyPart and I get the same problem if i put it as the
>>>>>>>>>>>>>>>>>>> first or last multipart.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's not a big deal, perhaps I should always use a
>>>>>>>>>>>>>>>>>>> different session code for concurrent operations but I'd like to be sure
>>>>>>>>>>>>>>>>>>> that we won't have this issue in production...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Good catch. Fixed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> There's a little mistake in the grizzly ahc provider
>>>>>>>>>>>>>>>>>>>> relative to the write queue size.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As you can see, the TransportCustomizer is called, and
>>>>>>>>>>>>>>>>>>>> then the write queue size (among other things) is set to AUTO_SIZE (instead
>>>>>>>>>>>>>>>>>>>> of previously UNLIMITED)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think the default settings like this AUTO_SIZE
>>>>>>>>>>>>>>>>>>>> attribute should be set before the customization of the transport, or they
>>>>>>>>>>>>>>>>>>>> would override the value we customized.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This is actually my case, since I can't reproduce my
>>>>>>>>>>>>>>>>>>>> "bug" which is "high memory consumption", even when using -1 / UNLIMITED in
>>>>>>>>>>>>>>>>>>>> the TransportCustomizer.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd
>>>>>>>>>>>>>>>>>>>> rather be able to tune this parameter during load tests to see the effect.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a
>>>>>>>>>>>>>>>>>>>>> 500m file with 40m heap size ;)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <
>>>>>>>>>>>>>>>>>>>>> ryan.lubke_at_oracle.com> a écrit :
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm going to be updating the Grizzly provider such
>>>>>>>>>>>>>>>>>>>>> that AUTO_SIZE (not AUTO_TUNE) is the default, so you can avoid the use of
>>>>>>>>>>>>>>>>>>>>> the TransportCustomizer.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regarding your tuning question, I would probably set
>>>>>>>>>>>>>>>>>>>>> the value to AsyncQueueWriter.AUTO_TUNE (this will be four times the socket
>>>>>>>>>>>>>>>>>>>>> write buffer) and see how that works.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> A question first. With these changes, your memory
>>>>>>>>>>>>>>>>>>>>> usage is more inline with what you were looking for?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will
>>>>>>>>>>>>>>>>>>>>> be released (with these new improvements?)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We would like to know if we wait for a release or if
>>>>>>>>>>>>>>>>>>>>> we install our own temp release on Nexus :)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If I let the initial value (-1) it won't block,
>>>>>>>>>>>>>>>>>>>>>> canWrite always returns true
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value
>>>>>>>>>>>>>>>>>>>>>> as a config attribute, neither the size of the write buffer you talked
>>>>>>>>>>>>>>>>>>>>>> about.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> So in the end, is there a way with current AHC code
>>>>>>>>>>>>>>>>>>>>>> to use this "canWrite = false" behavior?
>>>>>>>>>>>>>>>>>>>>>> If not, can you please provide a way to set this
>>>>>>>>>>>>>>>>>>>>>> behavior on v1.7.20 ? thanks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> PS: does it make sens to use the same number of bytes
>>>>>>>>>>>>>>>>>>>>>> un the feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Could you also provide a sample of how you're
>>>>>>>>>>>>>>>>>>>>>>> performing your feed?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1]
>>>>>>>>>>>>>>>>>>>>>>> and Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> By default, Grizzly will configure the async write
>>>>>>>>>>>>>>>>>>>>>>> queue length to be four times the write buffer size (which is based off the
>>>>>>>>>>>>>>>>>>>>>>> socket write buffer).
>>>>>>>>>>>>>>>>>>>>>>> When this queue exceeds this value, canWrite() will
>>>>>>>>>>>>>>>>>>>>>>> return false.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> When this occurs, you can register a WriteListener
>>>>>>>>>>>>>>>>>>>>>>> to be notified when the queue length is below the configured max and then
>>>>>>>>>>>>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>>>>>>>>>>>>> until the onWritePossible() callback has been
>>>>>>>>>>>>>>>>>>>>>>> invoked.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler()
>>>>>>>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>>>>> public void onWritePossible() throws
>>>>>>>>>>>>>>>>>>>>>>> Exception {
>>>>>>>>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>>>>>>>>>>>>> I use AHC default which seems to
>>>>>>>>>>>>>>>>>>>>>>> use SameThreadIOStrategy so I don't think it's related to the IO strategy.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of
>>>>>>>>>>>>>>>>>>>>>>> 50m, but I have to put a Thread.sleep(30) between each 100k Buffer send to
>>>>>>>>>>>>>>>>>>>>>>> the FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The connection with the server is not good here, but
>>>>>>>>>>>>>>>>>>>>>>> in production it is normally a lot better as far as I know.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I've tried things
>>>>>>>>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'd like the Grizzly internals to block when there
>>>>>>>>>>>>>>>>>>>>>>> are too much pending bytes to send. Is it possible?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> PS: I've just been able to send a 500mo file with
>>>>>>>>>>>>>>>>>>>>>>> 100mo heap, but it needed a sleep of 100ms between each 100k Buffer sent to
>>>>>>>>>>>>>>>>>>>>>>> the bodygenerator
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <
>>>>>>>>>>>>>>>>>>>>>>> lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner
>>>>>>>>>>>>>>>>>>>>>>>> used in the SSL filter?
>>>>>>>>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>>>>>>>>>>>>> I don't really understand why messages have to be
>>>>>>>>>>>>>>>>>>>>>>>> cloned, can I remove this? How?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <
>>>>>>>>>>>>>>>>>>>>>>>> lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a
>>>>>>>>>>>>>>>>>>>>>>>>> heap of 400m.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> In our real use cases we would probably have files
>>>>>>>>>>>>>>>>>>>>>>>>> under 20mo but we want to reduce the memory consumption because we can have
>>>>>>>>>>>>>>>>>>>>>>>>> x parallel uploads on the same server according to the user activity.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator
>>>>>>>>>>>>>>>>>>>>>>>>> reduced the memory footprint or if it's almost like before.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer
>>>>>>>>>>>>>>>>>>>>>>>>>> allocation is concerned, it's untunable.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> As to your second question, there is no suggested
>>>>>>>>>>>>>>>>>>>>>>>>>> size. This is all very application specific.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of
>>>>>>>>>>>>>>>>>>>>>>>>>> 33842 and it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so
>>>>>>>>>>>>>>>>>>>>>>>>>> that it consumes less memory?
>>>>>>>>>>>>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed
>>>>>>>>>>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each
>>>>>>>>>>>>>>>>>>>>>>>>>>> new buffer that is passed to the SSLFilter, we allocate a buffer twice the
>>>>>>>>>>>>>>>>>>>>>>>>>>> size in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an
>>>>>>>>>>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Depending on the socket configurations of both
>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints, and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>>>>>>>>>>>>> be the write queue is becoming too large. We
>>>>>>>>>>>>>>>>>>>>>>>>>>> do have a way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded
>>>>>>>>>>>>>>>>>>>>>>>>>>> here. I will see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded
>>>>>>>>>>>>>>>>>>>>>>>>>>> like before.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream
>>>>>>>>>>>>>>>>>>>>>>>>>>> the body or something like that?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull
>>>>>>>>>>>>>>>>>>>>>>>>>>>> request on AHC tomorrow with a better code using the same Part classes that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> already exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder
>>>>>>>>>>>>>>>>>>>>>>>>>>>> feeds the queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something, will work on that, it will be better than a thread sleep to slow
>>>>>>>>>>>>>>>>>>>>>>>>>>>> down the filepart reading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle adding something like this today. I'll follow up with something you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can test out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think I see what I could do, probably
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like that:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder = new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes =
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upload a file with AHC / Grizzly in streaming, I mean without loading the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whole file bytes in memory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> byte[] which contans the whole file, so it means that my server can be OOM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if too many users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> memory managers, with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wto the BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the file so that I could eventually consume only 20mo of heap while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> uploading a 500mo file?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the documentation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But if you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can be improved.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> receive a file with CXF and have to transmit it to a storage server (like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> S3). CXF doesn't consume memory bevause it is streaming the large fle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> uploads to the file system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>