Ok thanks :)
Take your time
We'll keep using no connection pool (as we used to do until now anyway, and
we could probably continue with this setting for a while because it passed
our load test goals)
2013/9/19 Ryan Lubke <ryan.lubke_at_oracle.com>
> Sorry, been under the weather. I'll be investigating this issue this
> morning.
>
> Stay tuned.
>
> Sébastien Lorber wrote:
>
> So what I've found is that:
>
>
> private final Connection.CloseListener listener = new
> Connection.CloseListener() {
> @Override
> public void onClosed(Connection connection,
> Connection.CloseType closeType) throws IOException {
> if (closeType == Connection.CloseType.REMOTELY) {
> if (LOG.isInfoEnabled()) {
> LOG.info("Remote closed connection ({}).
> Removing from cache", connection.toString());
> }
> }
> GrizzlyConnectionsPool.this.removeAll(connection);
> }
> };
>
>
> public boolean removeAll(Connection connection) {
>
> if (connection == null || closed.get()) {
> return false;
> }
> connection.removeCloseListener(listener);
> boolean isRemoved = false;
> for (Map.Entry<String, DelayedExecutor.IdleConnectionQueue> entry
> : connectionsPool.entrySet()) {
> boolean removed = entry.getValue().remove(connection);
> isRemoved |= removed;
> }
> return isRemoved;
>
> }
>
>
>
> When the connection is closed remotely, the connection that we try to
> remove is never in the cache, thus isRemoved = false
>
> I guess it has something to do with the "connection discrimination" you
> previously fixed and I didn't really understand :)
>
> Do you have an idea on this problem?
>
>
>
>
>
> By the way, I just tested again WITHOUT the FeedableBodyGenerator.
> I run batches of 10 concurrent uploads with a max 20 connection pool.
> I run these 10 concurrent uploads multiple times.
> It works fine, but if just after the request I add a Thread.sleep(15000)
> then it seems to lead to the same problem:
>
> @Test
> public void do_test_without_ahc_streaming() throws Exception {
> List<Response> responses = runConcurrently(CONCURRENT_UPLOADS,new
> Callable<Response>() {
> @Override
> public Response call() throws Exception {
> return uploadSingleDocumentWithoutStreaming();
> }
> });
> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
> @Override
> public Response call() throws Exception {
> return uploadSingleDocumentWithoutStreaming();
> }
> });
> Thread.sleep(15000);
> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
> @Override
> public Response call() throws Exception {
> return uploadSingleDocumentWithoutStreaming();
> }
> });
> Thread.sleep(15000);
> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
> @Override
> public Response call() throws Exception {
> return uploadSingleDocumentWithoutStreaming();
> }
> });
> Thread.sleep(15000);
> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
> @Override
> public Response call() throws Exception {
> return uploadSingleDocumentWithoutStreaming();
> }
> });
> Thread.sleep(15000);
> ......................................;
> }
>
> This permits to give some time for the remote host to close the connection
>
>
>
> So in the end it seems that we have troubles in any case when the remote
> host closes the connection
>
>
>
>
> Btw I just saw this issue:
> https://github.com/AsyncHttpClient/async-http-client/pull/311
> I checked the code that used AHC 1.7.7 and what I can see is that the
> connection pool is bypassed even with the property=true
> I have in my logs:
> [main] DEBUG
> com.ning.http.client.providers.grizzly.GrizzlyConnectionsPool:162 - [poll]
> No existing queue for uri [https://digiposte.orsid.com:443].
>
>
> So, by updating to 1.7.20-SNAPSHOT, the connection pool has been magically
> enabled, and it doesn't seem to work well for me.
>
>
> I'll keep that pool disabled because it's the behavior of our previous
> applicative version that is near to production and works pretty fine, but
> will open a Github issue because i'm quitting my job next month and they
> want to track when they'll be able to use a connection pool :)
>
>
>
> 2013/9/19 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>
>> So it seems I can reproduce this in local.
>>
>> Disabling the ssl connection pooling solve the problem.
>>
>>
>> We didn't have this problems with v1.7.7 we previously used.
>>
>> I just tested with the last commits of 1.7.20-SNAPSHOT, the problem is
>> still here.
>> The problem doesn't appear on normal AHC requests, neither on multipart
>> file upload when I do not use the FeedableBodyGenerator so it seems this is
>> the new behavior of FeedableBodyGenerator that has a problem with the ssl
>> connection pooling. Will try to investigate this.
>>
>> By the way, your last commits about selector/worker threads seems ok
>> (didn't notice any problem with the ssl pool disabled)
>>
>>
>>
>> 2013/9/18 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>
>>> Hi
>>>
>>> Unfortunatly it seems there's another problem.
>>> On our test environment I often get in the logs:
>>>
>>> Caught internal server errorjava.io.IOException: Maximum pooled
>>> connections exceeded
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.cleanup(GrizzlyAsyncHttpProvider.java:1415)
>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpPacketParsed(GrizzlyAsyncHttpProvider.java:1366)
>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>> at
>>> org.glassfish.grizzly.http.HttpCodecFilter.decodeWithTransferEncoding(HttpCodecFilter.java:1176)
>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:516)
>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:161)
>>> ~[grizzly-http-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> Wrapped by: java.util.concurrent.ExecutionException:
>>> java.io.IOException: Maximum pooled connections exceeded
>>> at
>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:359)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:265)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyResponseFuture.get(GrizzlyResponseFuture.java:165)
>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>>
>>>
>>>
>>>
>>> boolean canReturnConnection(final Connection c) {
>>>
>>> return (DO_NOT_CACHE.get(c) != null ||
>>> pool.canCacheConnection());
>>>
>>> }
>>>
>>>
>>> private static HttpTransactionContext cleanup(final
>>> FilterChainContext ctx,
>>> final
>>> GrizzlyAsyncHttpProvider provider) {
>>>
>>> final Connection c = ctx.getConnection();
>>> final HttpTransactionContext context =
>>> provider.getHttpTransactionContext(c);
>>> context.provider.setHttpTransactionContext(c, null);
>>> if
>>> (!context.provider.connectionManager.canReturnConnection(c)) {
>>> context.abort(new IOException("Maximum pooled
>>> connections exceeded"));
>>> } else {
>>> if
>>> (!context.provider.connectionManager.returnConnection(context.request, c)) {
>>> ctx.getConnection().close();
>>> }
>>> }
>>>
>>> return context;
>>>
>>> }
>>>
>>>
>>>
>>> We use:
>>>
>>> Key=[properties.httpclient.allowSslConnectionPool] Value=[true]
>>> Key=[properties.httpclient.maximumConnectionsPerHost] Value=[20]
>>> Key=[properties.httpclient.maximumConnectionsTotal] Value=[30]
>>> Key=[properties.httpclient.timeout.connection] Value=[10000]
>>> Key=[properties.httpclient.timeout.request] Value=[30000]
>>>
>>>
>>> In our logs, I can see there are 30 times this log:
>>>
>>> Remote closed connection (TCPNIOConnection{localSocketAddress={/
>>> 172.16.104.160:55488}, peerSocketAddress={host/172.16.4.100:443}}).
>>> Removing from cache
>>>
>>> and then it seems no connection is never available and the app is
>>> blocked.
>>>
>>> I'll try tomorrow to disable or increase the pool size, but if you have
>>> any idea of the problem please let me know.
>>> We do not run load tests, these are simple functional tests with nearly
>>> no concurrency.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>> 2013/9/17 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>
>>>> This is good to hear. Thanks for all the feedback and working with us
>>>> to nail this down.
>>>>
>>>>
>>>> Sébastien Lorber wrote:
>>>>
>>>> Thanks,
>>>>
>>>> We already installed the previous snapshot in our nexus because it
>>>> works fine and I'm working on something else now but I'll give you a
>>>> feedback soon to see if this still works fine.
>>>> For us it doesn't really mater which thread is using since we do not
>>>> use Future in our applications for this case.
>>>>
>>>>
>>>> Btw I've been able deploy a main with my code running concurrent
>>>> uploads on a server which has a better connectivity with the remote API and
>>>> it seems I can upload up to 150 concurrent files of 10mb with a heap of
>>>> 250mo in 30 seconds and a throughput near 50mb/s
>>>>
>>>> I don't really know the infrastructure on which my code was deployed
>>>> but I suspect Grizzly is not the bottleneck anymore :)
>>>> When I have more than 150 concurrent uploads the remote endpoint seems
>>>> to timeout under the load.
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/16 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>
>>>>>
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I noticed something strange.
>>>>> On the FileInputStream I have, I've added a log on the close() of the
>>>>> stream which is called once the whole file has been read to be sent to the
>>>>> feeder.
>>>>>
>>>>> 1) If I perform a request (ie my session init request in the previous
>>>>> discussions) before doing my multipart upload, the thread that does execute
>>>>> the feeding is the thread that fires the request, and not a Grizzly worker.
>>>>> [Thread-30] INFO Will close file stream
>>>>> java.io.FileInputStream_at_27fe4315 after having read 1440304 bytes
>>>>>
>>>>>
>>>>> 2) If I don't do any request before firing the multipart upload, the
>>>>> thread that does the feeding is a Grizzly threadpool worker thread:
>>>>> [Grizzly(22)] INFO Will close file stream
>>>>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>>>>
>>>>>
>>>>> Is this normal? I would expect a worker thread to always be used, and
>>>>> the main applicative thread that performs the request to never be blocking.
>>>>> (but it's not such an issue for me, we don't have a reactive non-blocking
>>>>> app anyway)
>>>>>
>>>>> It's currently expected behavior. Will need to re-evaluate this
>>>>> based on the new semantics of the FeedableBodyGenerator.
>>>>>
>>>>> I've committed a change for this. If, upon testing, you find there is
>>>>> still an issue, please let us know.
>>>>>
>>>>>
>>>>>
>>>>> On the 1st case, here's the stacktrace when the feed method is called:
>>>>>
>>>>> * at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)
>>>>> *
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>> at
>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
>>>>> at
>>>>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>>>>> at
>>>>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
>>>>> * at
>>>>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)
>>>>> *
>>>>>
>>>>> So it seems this happen when the handshake has already been done when
>>>>> *initializeAsynchronousTransfer* is called, so that we do not go
>>>>> through the HandshakeListener
>>>>>
>>>>>
>>>>> Notice that this case seems to also happen in a few threads on the 2nd
>>>>> case, but most of the threads are Grizzly workers.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>>
>>>>>> Thanks, it seems to work.
>>>>>>
>>>>>> I would suggest to throw IOException on the flush method since the
>>>>>> feed method is supposed to be called here
>>>>>>
>>>>>>
>>>>>> My implementation of flush is:
>>>>>>
>>>>>> @Override
>>>>>> public void flush() {
>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>> try ( OutputStream outputStream = createFeedingOutputStream() )
>>>>>> {
>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>> } catch (Exception e) {
>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>> FeedableBodyGenerator",e);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Is this correct? The OutputStream redirects the bytes written to the
>>>>>> feed(Buffer) method
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> There seems to be some concurrency issue. Because the upload of 1
>>>>>> file seems fine, but when using multiple threads, I often get the following
>>>>>> stack:
>>>>>> Caused by: java.io.IOException: Stream Closed
>>>>>> at java.io.FileInputStream.readBytes(Native Method)
>>>>>> at java.io.FileInputStream.read(FileInputStream.java:242)
>>>>>> at
>>>>>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>
>>>>>>
>>>>>> This is because the flush() method is called multiple times for the
>>>>>> same request on some cases.
>>>>>> I guess this is not supposed to happen.
>>>>>> What I understand is that the flush() method is supposed to be called
>>>>>> only once.
>>>>>>
>>>>>>
>>>>>> Using debug logging breakpoints I get the following:
>>>>>>
>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18 started
>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19 started
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>>>>>> Adding handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>> *Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>> *
>>>>>> *Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>> *
>>>>>> *Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>> *
>>>>>> *Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>> *
>>>>>> *Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>> *
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>> Completing and removing handshake listener for
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>>>>>
>>>>>>
>>>>>> As you can see the same HandshakeListener seems to be called multiple
>>>>>> times for the same FeedableBodyGenerator
>>>>>>
>>>>>> When this happens, the stack is:
>>>>>>
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So I tried with the following code:
>>>>>>
>>>>>> private boolean alreadyFlushed = false;
>>>>>> @Override
>>>>>> public synchronized void flush() {
>>>>>> if ( alreadyFlushed ) {
>>>>>> return;
>>>>>> }
>>>>>> startFeeding();
>>>>>> alreadyFlushed = true;
>>>>>> }
>>>>>>
>>>>>> It works fine when upload small files.
>>>>>>
>>>>>> But with larger files, I often get TimeoutException stacks for some
>>>>>> of the threads:
>>>>>>
>>>>>> Caused by: java.util.concurrent.TimeoutException
>>>>>> at
>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>> at
>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Did I miss something?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>
>>>>>>> Committed another small change. Please make sure you're at the
>>>>>>> latest when you build.
>>>>>>>
>>>>>>> -rl
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>
>>>>>>> Okay, I've committed another set of refactorings to the
>>>>>>> FeedableBodyGenerator.
>>>>>>>
>>>>>>> For your use case, you should extend
>>>>>>> FeedableBodyGenerator.SimpleFeeder.
>>>>>>>
>>>>>>> Let me know if you run into issues.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>>
>>>>>>> Yes I think it would, so that I can feed the queue at once
>>>>>>>
>>>>>>> One thread will be locked during the feeding for nothing but it's
>>>>>>> not a real problem in my usecase.
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>> Would having a different listener that will be notified once async
>>>>>>>> transferring has been started work better for you?
>>>>>>>>
>>>>>>>> Something like:
>>>>>>>>
>>>>>>>> onAsyncTransferInitiated() {
>>>>>>>> // invoke your feed method
>>>>>>>> }
>>>>>>>>
>>>>>>>> ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>
>>>>>>>> Unfortunatly I won't be able to use the Feeder non-blocking stuff
>>>>>>>> for now, because of how the multipart request in handled in AHC
>>>>>>>>
>>>>>>>>
>>>>>>>> Here's my feeding method:
>>>>>>>>
>>>>>>>> public void feed() throws IOException {
>>>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>>>> try ( OutputStream outputStream = createFeedingOutputStream() )
>>>>>>>> {
>>>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>>>> } catch (Exception e) {
>>>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>>>> FeedableBodyGenerator",e);
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> As you can see, the multipart Parts array can only be pushed to the
>>>>>>>> OutputStream, I don't have any way to "pull" the data when the canFeed()
>>>>>>>> method is triggered.
>>>>>>>>
>>>>>>>>
>>>>>>>> But I've seen that there's
>>>>>>>> a com.ning.http.multipart.MultipartBody#read that seems to provide a memory
>>>>>>>> efficient way to pull data from a Multipart body...
>>>>>>>>
>>>>>>>> Should see what I come up with this
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>
>>>>>>>>> It seems the Feeder is highly recommended but not mandatory so I
>>>>>>>>> tried without.
>>>>>>>>>
>>>>>>>>> With my existing code it seems there is a synchronization problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The feeding threads get locked to the prematureFeed.get();
>>>>>>>>>
>>>>>>>>> So the Grizzly kernel threads are unable to acquire the lock
>>>>>>>>> required to enter the initializeAsynchronousTransfer method
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Will try with an implementation of Feeder
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>
>>>>>>>>>> Hmmm it seems I have a problem with one of your maven plugins.
>>>>>>>>>> I'll try to bypass it, but for info:
>>>>>>>>>>
>>>>>>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>>>>>>> [WARNING]
>>>>>>>>>> [WARNING] Some problems were encountered while building the
>>>>>>>>>> effective settings
>>>>>>>>>> [WARNING] 'profiles.profile[default].repositories.repository.id'
>>>>>>>>>> must be unique but found duplicate repository with id
>>>>>>>>>> fullsix-maven-repository @ /home/slorber/.m2/settings.xml
>>>>>>>>>> [WARNING]
>>>>>>>>>> [INFO] Scanning for projects...
>>>>>>>>>> [INFO]
>>>>>>>>>>
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>>>>>>>>> async-http-client ---
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>>>>>>>>>> (enforce-maven) @ async-http-client ---
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>>>>>>>>>> (enforce-versions) @ async-http-client ---
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] --- maven-resources-plugin:2.4.3:resources
>>>>>>>>>> (default-resources) @ async-http-client ---
>>>>>>>>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>>>>>>> [INFO]
>>>>>>>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile)
>>>>>>>>>> @ async-http-client ---
>>>>>>>>>> [INFO] Compiling 158 source files to
>>>>>>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>>>>>>> [INFO]
>>>>>>>>>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>>>>>>>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>>>>>>>> *[INFO] Checking unresolved references to
>>>>>>>>>> org.codehaus.mojo.signature:java15:1.0*
>>>>>>>>>> *[ERROR] Undefined reference:
>>>>>>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>>>>>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class
>>>>>>>>>> *
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> [INFO] Total time: 8.747s
>>>>>>>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>>>>>>>> [INFO] Final Memory: 30M/453M
>>>>>>>>>> [INFO]
>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>> *[ERROR] Failed to execute goal
>>>>>>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>>>>>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>>>>>>>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]
>>>>>>>>>> *
>>>>>>>>>> [ERROR]
>>>>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
>>>>>>>>>> with the -e switch.
>>>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
>>>>>>>>>> logging.
>>>>>>>>>> [ERROR]
>>>>>>>>>> [ERROR] For more information about the errors and possible
>>>>>>>>>> solutions, please read the following articles:
>>>>>>>>>> [ERROR] [Help 1]
>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Ok thank you, I'll try to implement that today and will give you
>>>>>>>>>>> my feedback :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>> Okay,
>>>>>>>>>>>>
>>>>>>>>>>>> I've committed my initial changes to the AHC repository.
>>>>>>>>>>>> Here's a summary of the changes:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>>>>>>>>> - Don't allow queueing of data before initiateAsyncTransfer has
>>>>>>>>>>>> been invoked. In low memory
>>>>>>>>>>>> heaps, this could lead to an OOM if the source is feeding too
>>>>>>>>>>>> fast. The new behavior is to
>>>>>>>>>>>> block until initiateAsyncTransfer is called, at which time the
>>>>>>>>>>>> blocked thread may proceed with
>>>>>>>>>>>> the feed operation.
>>>>>>>>>>>> - Introduce the concept of a Feeder. Implementations are
>>>>>>>>>>>> responsible, at a high level, for:
>>>>>>>>>>>> + letting the provider know that data is available to be fed
>>>>>>>>>>>> without blocking
>>>>>>>>>>>> + allowing the registration of a callback that the Feeder
>>>>>>>>>>>> implementation may invoke
>>>>>>>>>>>> to signal that more data is available, if it wasn't available
>>>>>>>>>>>> at a previous point in time.
>>>>>>>>>>>> - When using a Feeder with a secure request, the SSL handshake
>>>>>>>>>>>> will be kicked off by the
>>>>>>>>>>>> initiateAsyncTransfer call, but feeding of data will not occur
>>>>>>>>>>>> until the handshake is complete.
>>>>>>>>>>>> This is necessary as the SSLFilter will queue up all writes
>>>>>>>>>>>> until the handshake is complete,
>>>>>>>>>>>> and currently, the buffer isn't tied in with the transport flow
>>>>>>>>>>>> control mechanism.
>>>>>>>>>>>> NOTE: This new SSL behavior is not currently applied when
>>>>>>>>>>>> invoking the feed() method
>>>>>>>>>>>> outside the context of a Feeder. Still need to address that.
>>>>>>>>>>>> - Exposed configuration of the async write queue limit through
>>>>>>>>>>>> the FeedableBodyGenerator.
>>>>>>>>>>>> This is an improvement on using a TransportCustomizer as any
>>>>>>>>>>>> configuration there is
>>>>>>>>>>>> transport-wide, and therefor applied to all Connections. By
>>>>>>>>>>>> exposing it here, each feeder
>>>>>>>>>>>> may have a different byte limit.
>>>>>>>>>>>> - Improved documentation for this class*
>>>>>>>>>>>>
>>>>>>>>>>>> I recommend reading through the javadoc comments in the source
>>>>>>>>>>>> [1] for FeedableBodyGenerator (comments welcome).
>>>>>>>>>>>> Additionally, I would re-work your code to leverage the Feeder
>>>>>>>>>>>> instead of calling feed() directly.
>>>>>>>>>>>>
>>>>>>>>>>>> If you have issues implementing Feeder, do let us know.
>>>>>>>>>>>>
>>>>>>>>>>>> If you have additional questions, again, let us know.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> -rl
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> So in the end I've end up with an implementation that's working
>>>>>>>>>>>> for me.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think there are 2 bugs:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue
>>>>>>>>>>>> if the initialize(ctx) method is not called fast enough.
>>>>>>>>>>>> This can be solved by using a BlockingQueue of size 1 and the
>>>>>>>>>>>> put() method.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Once the context is injected, the FeedableBodyGenerator
>>>>>>>>>>>> flushes the queue.
>>>>>>>>>>>> The matter is that if the connection is new, not warmed up by a
>>>>>>>>>>>> previous request, then the SSL handshake is not done yet, and it seems that
>>>>>>>>>>>> the bytes are accumulated in some part of the SSL filter which doesn't
>>>>>>>>>>>> deliver them to the connection until the handshake has completed,
>>>>>>>>>>>> so c.canWrite() continues to return true.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have replaced some part of the FeedableBodyGenerator to test
>>>>>>>>>>>> this and it works pretty fine. See what I have changed:
>>>>>>>>>>>>
>>>>>>>>>>>> 1)
>>>>>>>>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>>>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2)
>>>>>>>>>>>> public void feed(final Buffer buffer, final boolean last)
>>>>>>>>>>>> throws IOException {
>>>>>>>>>>>> try {
>>>>>>>>>>>> queue.put(new BodyPart(buffer, last));
>>>>>>>>>>>> } catch (InterruptedException e) {
>>>>>>>>>>>> throw new RuntimeException(e);
>>>>>>>>>>>> }
>>>>>>>>>>>> queueSize.incrementAndGet();
>>>>>>>>>>>>
>>>>>>>>>>>> if (context != null) {
>>>>>>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>>> }
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> private void
>>>>>>>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>>>>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>>>>>>> try { Thread.sleep(10); } catch ( Exception e ) {
>>>>>>>>>>>> throw new RuntimeException(e); }
>>>>>>>>>>>> }
>>>>>>>>>>>> }
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> private boolean connectionIsReadyToWrite(FilterChainContext
>>>>>>>>>>>> fcc) {
>>>>>>>>>>>> Connection connection = fcc.getConnection();
>>>>>>>>>>>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>>>>>>>>>>>> return sslEngine != null &&
>>>>>>>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We had come to similar conclusions on this end. I'm still
>>>>>>>>>>>> working through testing the idea I mentioned previously (took longer than I
>>>>>>>>>>>> expected - sorry).
>>>>>>>>>>>> I hope to have something for you to test very soon.
>>>>>>>>>>>>
>>>>>>>>>>>> Note that it will be taking the above into account as well.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried to put a while ( context == null ) Thread.sleep
>>>>>>>>>>>>> but it doesn't seem to work, when the context gets injected, after the
>>>>>>>>>>>>> sleeps, there's an OOM
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have done another test, remember my code that worked, which
>>>>>>>>>>>>> previously "warmed" the Thread with an useless request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> private Runnable uploadSingleDocumentRunnable = new
>>>>>>>>>>>>> Runnable() {
>>>>>>>>>>>>> @Override
>>>>>>>>>>>>> public void run() {
>>>>>>>>>>>>> try {
>>>>>>>>>>>>> getUselessSessionCode();
>>>>>>>>>>>>> Thread.sleep(X);
>>>>>>>>>>>>> uploadSingleDocument();
>>>>>>>>>>>>> } catch ( Exception e ) {
>>>>>>>>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>> };
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have put a sleep of X between the useless warmup request,
>>>>>>>>>>>>> and the real upload request
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What I noticed is that there is a very different behavior
>>>>>>>>>>>>> according to the value of X
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Under 10 seconds, it seems the stuff is still warm, I can
>>>>>>>>>>>>> upload the documents.
>>>>>>>>>>>>> Around 10 seconds I get a stack which seems to be "connection
>>>>>>>>>>>>> closed" or something
>>>>>>>>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> The stacks I get for 10 seconds looks like
>>>>>>>>>>>>>
>>>>>>>>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I got some other different stacks saying Connection
>>>>>>>>>>>>> Closed Remotely or something like that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> So it seems that something is bound to my thread, and it stays
>>>>>>>>>>>>> bound to it for about 10 seconds, do you have any idea what it could be?
>>>>>>>>>>>>> (My connection timeout setting seems to have no effect on this
>>>>>>>>>>>>> 10s threshold)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> That is one solution. I'm working out an alternative right
>>>>>>>>>>>>>> now. Stay tuned!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyway it's not a problem, I think the
>>>>>>>>>>>>>> FeedableBodyGenerator.feed() method just has to block until a context has
>>>>>>>>>>>>>> been (and ThreadCache initialized) to avoid OOM errors
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What is very strange is that I tested with/without the same
>>>>>>>>>>>>>>> sessionCode with our previous code, the one not using
>>>>>>>>>>>>>>> FeedableBodyGenerator, which has a high memory consumption.
>>>>>>>>>>>>>>> Despites the fact it had high memory consumption, it seems
>>>>>>>>>>>>>>> work fine to upload multiple documents if allocated with a large heap, and
>>>>>>>>>>>>>>> the sessionCode seems to have no effect.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On the new impl using the FeedableBodyGenerator, the
>>>>>>>>>>>>>>> sessionCode sent as a multipart bodypart seems to have an effect.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have tried to feed the queue before sending the request to
>>>>>>>>>>>>>>> AHC, but this leads to this exception (with/without sessionCode switching)
>>>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout
>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By the way, by using a low timeout with the same
>>>>>>>>>>>>>>>> sessioncode, I got the following NPE:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>>>>>>>>>> httpCtx =
>>>>>>>>>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is right, here's a log I have when I use the same
>>>>>>>>>>>>>>>>> session code, ie the remote host is blocking the data or something.
>>>>>>>>>>>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> *Flushing queue of size 97 with allowBlocking = false*
>>>>>>>>>>>>>>>>> *Flushing queue of size 100 with allowBlocking = false*
>>>>>>>>>>>>>>>>> *Flushing queue of size 160 with allowBlocking = true*
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>>>>>>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268',
>>>>>>>>>>>>>>>>> transport: 'socket'
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Otherwise, with different session codes, I get the
>>>>>>>>>>>>>>>>> following:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>> *Flushing queue of size 0 with allowBlocking = false*
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>>>>>>>>>>>> ... and this continues without OOM
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, this seems to be the problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it would be great to be able to be able to choose
>>>>>>>>>>>>>>>>> the queue impl behind that FeedableBodyGenerator, like I suggested in my
>>>>>>>>>>>>>>>>> pull request.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> See here:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in
>>>>>>>>>>>>>>>>> this context, and in my case I would probably use a queue of size 1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This would handle the blocking of the feed method, without
>>>>>>>>>>>>>>>>> having to use this:
>>>>>>>>>>>>>>>>> if (context != null) {
>>>>>>>>>>>>>>>>> flushQueue(true);
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Or perhaps the feed() method have to wait until a context
>>>>>>>>>>>>>>>>> is set in the BodyGenerator ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it would be more clear if
>>>>>>>>>>>>>>>>> the initializeAsynchronousTransfer simply didn't flush the queue but just
>>>>>>>>>>>>>>>>> setup the context.
>>>>>>>>>>>>>>>>> Then the feed method would block until there's a context
>>>>>>>>>>>>>>>>> set, and then flush the queue with blocking behavior.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is probably the next step, but as we are using AHC
>>>>>>>>>>>>>>>>> for async, it would probably be great if that blocking feed() method was
>>>>>>>>>>>>>>>>> called in a worker thread instead of our main thread.
>>>>>>>>>>>>>>>>> I won't use this but someone who really wants a
>>>>>>>>>>>>>>>>> non-blocking impl of performant multipart fileupload would probably need
>>>>>>>>>>>>>>>>> this, or will use an ExecutorService for the feeding operations as a
>>>>>>>>>>>>>>>>> workaround.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks again for your reactivity
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've integrated this change and it works fine except a
>>>>>>>>>>>>>>>>>> little detail.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>>>>>>>>>>>>> This API requires a "sessionCode" in each request. So
>>>>>>>>>>>>>>>>>> there is a multipart StringPart with that SessionCode.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We used to have a cache which holds the sessionCode 30min
>>>>>>>>>>>>>>>>>> per user so that we do not need to init a new session each time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I had troubles in this very specific case: when I upload
>>>>>>>>>>>>>>>>>> 5 docs with the same session code.
>>>>>>>>>>>>>>>>>> When I remove the cache and use 5 different session
>>>>>>>>>>>>>>>>>> codes, it works fine.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I guess the remote service is blocking concurrent uploads
>>>>>>>>>>>>>>>>>> with the same session code. I don't know at all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Where I want to go is that I wouldn't have expected
>>>>>>>>>>>>>>>>>> Grizzly to OOM
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>>>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>>>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of
>>>>>>>>>>>>>>>>>> each Buffer sent to the FeedableBodyGenerator
>>>>>>>>>>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>>>>>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do you have any idea why is there an OOM with these
>>>>>>>>>>>>>>>>>> settings?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Perhaps it is because the feed() method of
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator doesn't block until the context is initialized.
>>>>>>>>>>>>>>>>>> I guess the initializeAsynchronousTransfer is called only
>>>>>>>>>>>>>>>>>> once the connection is established, and perhaps a lot of Buffer are added
>>>>>>>>>>>>>>>>>> to the queue...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, it's invoked once the request has been dispatched,
>>>>>>>>>>>>>>>>>> so if the generator is fed a lot before the request, I could see this
>>>>>>>>>>>>>>>>>> happening.
>>>>>>>>>>>>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But I'm not sure at all because the session code is
>>>>>>>>>>>>>>>>>> transmitted as a BodyPart and I get the same problem if i put it as the
>>>>>>>>>>>>>>>>>> first or last multipart.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It's not a big deal, perhaps I should always use a
>>>>>>>>>>>>>>>>>> different session code for concurrent operations but I'd like to be sure
>>>>>>>>>>>>>>>>>> that we won't have this issue in production...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Good catch. Fixed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There's a little mistake in the grizzly ahc provider
>>>>>>>>>>>>>>>>>>> relative to the write queue size.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As you can see, the TransportCustomizer is called, and
>>>>>>>>>>>>>>>>>>> then the write queue size (among other things) is set to AUTO_SIZE (instead
>>>>>>>>>>>>>>>>>>> of previously UNLIMITED)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think the default settings like this AUTO_SIZE
>>>>>>>>>>>>>>>>>>> attribute should be set before the customization of the transport, or they
>>>>>>>>>>>>>>>>>>> would override the value we customized.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is actually my case, since I can't reproduce my
>>>>>>>>>>>>>>>>>>> "bug" which is "high memory consumption", even when using -1 / UNLIMITED in
>>>>>>>>>>>>>>>>>>> the TransportCustomizer.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This could work fine for me with AUTO_SIZE, but I'd
>>>>>>>>>>>>>>>>>>> rather be able to tune this parameter during load tests to see the effect.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks i will ckeck that on monday. I can now upload a
>>>>>>>>>>>>>>>>>>>> 500m file with 40m heap size ;)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Envoyé de mon iPhone
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <
>>>>>>>>>>>>>>>>>>>> ryan.lubke_at_oracle.com> a écrit :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm going to be updating the Grizzly provider such that
>>>>>>>>>>>>>>>>>>>> AUTO_SIZE (not AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>>>>>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding your tuning question, I would probably set
>>>>>>>>>>>>>>>>>>>> the value to AsyncQueueWriter.AUTO_TUNE (this will be four times the socket
>>>>>>>>>>>>>>>>>>>> write buffer) and see how that works.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> A question first. With these changes, your memory
>>>>>>>>>>>>>>>>>>>> usage is more inline with what you were looking for?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> By the way, do you have any idea when the 1.7.20 will
>>>>>>>>>>>>>>>>>>>> be released (with these new improvements?)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We would like to know if we wait for a release or if we
>>>>>>>>>>>>>>>>>>>> install our own temp release on Nexus :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you, it works fine!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I just had to modify a single line after your commit.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If I let the initial value (-1) it won't block,
>>>>>>>>>>>>>>>>>>>>> canWrite always returns true
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Btw, on AHC I didn't find any way to pass this value
>>>>>>>>>>>>>>>>>>>>> as a config attribute, neither the size of the write buffer you talked
>>>>>>>>>>>>>>>>>>>>> about.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So in the end, is there a way with current AHC code to
>>>>>>>>>>>>>>>>>>>>> use this "canWrite = false" behavior?
>>>>>>>>>>>>>>>>>>>>> If not, can you please provide a way to set this
>>>>>>>>>>>>>>>>>>>>> behavior on v1.7.20 ? thanks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> PS: does it make sens to use the same number of bytes
>>>>>>>>>>>>>>>>>>>>> un the feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>>>>>>>> ? do you have any tuning recommandation?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Please disregard.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Could you also provide a sample of how you're
>>>>>>>>>>>>>>>>>>>>>> performing your feed?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ryan Lubke wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'd recommend looking at Connection.canWrite() [1]
>>>>>>>>>>>>>>>>>>>>>> and Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> By default, Grizzly will configure the async write
>>>>>>>>>>>>>>>>>>>>>> queue length to be four times the write buffer size (which is based off the
>>>>>>>>>>>>>>>>>>>>>> socket write buffer).
>>>>>>>>>>>>>>>>>>>>>> When this queue exceeds this value, canWrite() will
>>>>>>>>>>>>>>>>>>>>>> return false.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> When this occurs, you can register a WriteListener to
>>>>>>>>>>>>>>>>>>>>>> be notified when the queue length is below the configured max and then
>>>>>>>>>>>>>>>>>>>>>> simulate blocking
>>>>>>>>>>>>>>>>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> final FutureImpl<Boolean> future =
>>>>>>>>>>>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> // Connection may be obtained by calling
>>>>>>>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>>>> public void onWritePossible() throws
>>>>>>>>>>>>>>>>>>>>>> Exception {
>>>>>>>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>>>>>>> public void onError(Throwable t) {
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>>>>> final long writeTimeout = 30;
>>>>>>>>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>>>>>>>> } catch (ExecutionException e) {
>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext httpCtx =
>>>>>>>>>>>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ryan, I've did some other tests.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It seems that using a blocking queue in the
>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator is totally useless because the thread consuming it is
>>>>>>>>>>>>>>>>>>>>>> not blocking and the queue never blocks the feeding, which was my intention
>>>>>>>>>>>>>>>>>>>>>> in the beginning. Maybe it depends on the IO strategy used?
>>>>>>>>>>>>>>>>>>>>>> I use AHC default which seems to
>>>>>>>>>>>>>>>>>>>>>> use SameThreadIOStrategy so I don't think it's related to the IO strategy.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> So in the end I can upload a 70m file with a heap of
>>>>>>>>>>>>>>>>>>>>>> 50m, but I have to put a Thread.sleep(30) between each 100k Buffer send to
>>>>>>>>>>>>>>>>>>>>>> the FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The connection with the server is not good here, but
>>>>>>>>>>>>>>>>>>>>>> in production it is normally a lot better as far as I know.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I've tried things
>>>>>>>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>>>>>>>> but it doesn't seem to work for me.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'd like the Grizzly internals to block when there
>>>>>>>>>>>>>>>>>>>>>> are too much pending bytes to send. Is it possible?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> PS: I've just been able to send a 500mo file with
>>>>>>>>>>>>>>>>>>>>>> 100mo heap, but it needed a sleep of 100ms between each 100k Buffer sent to
>>>>>>>>>>>>>>>>>>>>>> the bodygenerator
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <
>>>>>>>>>>>>>>>>>>>>>> lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> By chance do you if I can remove the MessageCloner
>>>>>>>>>>>>>>>>>>>>>>> used in the SSL filter?
>>>>>>>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>>>>>>>>>>>>>>>>> I don't really understand why messages have to be
>>>>>>>>>>>>>>>>>>>>>>> cloned, can I remove this? How?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/8/29 Sébastien Lorber <
>>>>>>>>>>>>>>>>>>>>>>> lorber.sebastien_at_gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'm trying to send a 500m file for my tests with a
>>>>>>>>>>>>>>>>>>>>>>>> heap of 400m.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> In our real use cases we would probably have files
>>>>>>>>>>>>>>>>>>>>>>>> under 20mo but we want to reduce the memory consumption because we can have
>>>>>>>>>>>>>>>>>>>>>>>> x parallel uploads on the same server according to the user activity.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'll try to check if using this BodyGenerator
>>>>>>>>>>>>>>>>>>>>>>>> reduced the memory footprint or if it's almost like before.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> At this point in time, as far as the SSL buffer
>>>>>>>>>>>>>>>>>>>>>>>>> allocation is concerned, it's untunable.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> As to your second question, there is no suggested
>>>>>>>>>>>>>>>>>>>>>>>>> size. This is all very application specific.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I have seen a lot of buffers which have a size of
>>>>>>>>>>>>>>>>>>>>>>>>> 33842 and it seems the limit is near half the capacity.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Perhaps there's a way to tune that buffer size so
>>>>>>>>>>>>>>>>>>>>>>>>> that it consumes less memory?
>>>>>>>>>>>>>>>>>>>>>>>>> Is there an ideal Buffer size to send to the feed
>>>>>>>>>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the OOM: as it stands now, for each new
>>>>>>>>>>>>>>>>>>>>>>>>>> buffer that is passed to the SSLFilter, we allocate a buffer twice the size
>>>>>>>>>>>>>>>>>>>>>>>>>> in order to
>>>>>>>>>>>>>>>>>>>>>>>>>> accommodate the encrypted result. So there's an
>>>>>>>>>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Depending on the socket configurations of both
>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints, and how fast the remote is reading data, it could
>>>>>>>>>>>>>>>>>>>>>>>>>> be the write queue is becoming too large. We do
>>>>>>>>>>>>>>>>>>>>>>>>>> have a way to detect this situation, but I'm pretty sure
>>>>>>>>>>>>>>>>>>>>>>>>>> the Grizzly internals are currently shielded
>>>>>>>>>>>>>>>>>>>>>>>>>> here. I will see what I can do to allow users to leverage this.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> With my usecase it works, the file is uploaded
>>>>>>>>>>>>>>>>>>>>>>>>>> like before.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Is it possible that SSL doesn't allow to stream
>>>>>>>>>>>>>>>>>>>>>>>>>> the body or something like that?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>>>>>>>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull
>>>>>>>>>>>>>>>>>>>>>>>>>>> request on AHC tomorrow with a better code using the same Part classes that
>>>>>>>>>>>>>>>>>>>>>>>>>>> already exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I created an OutputStream that redirects to the
>>>>>>>>>>>>>>>>>>>>>>>>>>> BodyGenerator feeder.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The problem I currently have is that the feeder
>>>>>>>>>>>>>>>>>>>>>>>>>>> feeds the queue faster than the async thread polling it :)
>>>>>>>>>>>>>>>>>>>>>>>>>>> I need to expose a limit to that queue size or
>>>>>>>>>>>>>>>>>>>>>>>>>>> something, will work on that, it will be better than a thread sleep to slow
>>>>>>>>>>>>>>>>>>>>>>>>>>> down the filepart reading
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, something like that. I was going to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle adding something like this today. I'll follow up with something you
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can test out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think I see what I could do, probably
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like that:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder = new MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Request uploadRequest1 = new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response> asyncRes =
>>>>>>>>>>>>>>>>>>>>>>>>>>>> asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>>>>>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to know if it's possible to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upload a file with AHC / Grizzly in streaming, I mean without loading the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whole file bytes in memory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The default behavior seems to allocate a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> byte[] which contans the whole file, so it means that my server can be OOM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if too many users upload a large file in the same time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managers, with reallocate=true/false but no more success.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems the whole file content is appended
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wto the BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> At least this seems to be the case with AHC
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> file so that I could eventually consume only 20mo of heap while uploading a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 500mo file?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I didn't notice anything related to that in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the documentation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But if you're tied to using Multipart uploads, you'd have to convert the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multipart data to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll take a closer look to see if this area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can be improved.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a file with CXF and have to transmit it to a storage server (like S3). CXF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't consume memory bevause it is streaming the large fle uploads to the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> file system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>