Hi Ryan and Sébastien,
I met the same issue(OOM) while I sent the large file(about 1G size) with
multipart in AHC.
When I reviewed AHC's sources, I couldn't find MultipartBodyGeneratorFeeder
class yet.
Is there anypdates? or where can I get the stable reference?
Thank in advance!
Regards,
Bongjae Chang
From: Sébastien Lorber <lorber.sebastien_at_gmail.com>
Reply-To: <users_at_grizzly.java.net>
De: Wednesday, September 25, 2013 5:47 PM
To: "users_at_grizzly.java.net" <users_at_grizzly.java.net>
Subject: Re: Upload a large file without oom with Grizzly
Thanks Ryan, will use that release :)
No problem until now, will tell you what happens during our load tests
2013/9/24 Ryan Lubke <ryan.lubke_at_oracle.com>
> Sébastien,>
> AHC 1.7.20 has been released. It's up on the oss.sonatype.org
> <http://oss.sonatype.org> maven repository and will be sync'd with central
> soon-ish.
>
> -rl
>
> Ryan Lubke wrote:
>>
>>
>> Sébastien Lorber wrote:
>>> Yes it works nicely :)
>>>
>>> Thanks Ryan I think everything is now working for me
>>>
>> You're welcome. Let us kno if you run into anything else, or if you have
> ideas on improvements, etc.
>>
>>
>>>
>>>
>>> 2013/9/20 Ryan Lubke <yan.lubke_at_oracle.com>
>>>> Just following up for closure here. Pool issue has ben confirmed as
>>>> resolved.
>>>>
>>>> Sébastien Lorber wrote:
>>>>> Ok thanks :)
>>>>> >>>>> Take your time
>>>>> We'll keep using no connection pool (as we used to do until now anyway,
>>>>> and we could probably continue with this setting for a while because it
>>>>> passed our load test goals)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/19 Ran Lubke <ryan.lubke_at_oracle.com>
>>>>>> Sorry, been under the weather. I'll e investigating this issue this
>>>>>> morning.
>>>>>>
>>>>>> Stay tuned.
>>>>>>
>>>>>> Sébastien Lorbr wrote:
>>>>>>> So what I've found is that:
>>>>>>>
>>>>>>>
>>>>>>> prvate final Connection.CloseListener listener = new
>>>>>>> Connection.CloseListener() {
>>>>>>> @Override
>>>>>>> public void onClosed(Connection connection,
>>>>>>> Connection.CloseType closeType) throws IOException {
>>>>>>> if (closeType == Connection.CloseType.EMOTELY) {
>>>>>>> if (LOG.isInfoEnabled()) {
>>>>>>> LOG.info("Remote closedconnection ({}).
>>>>>>> Removing from cache", connection.toString());
>>>>>>> }
>>>>>>> }
>>>>>>> GrizzlyConnectionsPool.this.removeAll(connection);
>>>>>>> }
>>>>>>> };
>>>>>>>
>>>>>>>
>>>>>>> public boolean removeAll(Connection connection) {
>>>>>>
>>>>>>> if (connection == null || closed.get()) {
>>>>>>> return false;
>>>>>>> }
>>>>>>> connection.removeCloseListener(listener);
>>>>>>> boolean isRemoved = false;
>>>>>>> for (Map.Entry<String, DelayedExecutor.IdleConnectionQueue>
>>>>>>> entry : connectionsPool.entrySet()) {
>>>>>>> oolean removed = entry.getValue().remove(connection);
>>>>>>> isReoved |= removed;
>>>>>>> }
>>>>>>> return isRemoved;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> When the connection is closed remotely, the connection that we try to>>>>>>> remove is never in the cache, thus isRemoved = false
>>>>>>>
>>>>>>> I guess it has something to do with the "connecion discrimination" you
>>>>>>> previously fixed and I didn't really understand :)
>>>>>>>
>>>>>>> Do you have an idea on this problem?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> By the way, I just tested again WITHOUT the FeedableBodyGenerator.
>>>>>>> I run batches of 10 concurrent uloads with a max 20 connection pool.
>>>>>>> I run these 10 concurrent uploads multiple times.
>>>>>>> It works fine, but if just after the request I add a Thread.sleep(15000)
>>>>>>> then it seems to lead to the same problem:
>>>>>>>
>>>>>> @Test
>>>>>>> public void do_test_without_ahc_streaming() hrows Exception {
>>>>>>> List<Response> responses = runConcrrently(CONCURRENT_UPLOADS,new
>>>>>>> Callable<Response>() {
>>>>>>> @Override
>>>>>>> public Response call() throws Exception {
>>>>>>> return uploadSingleDocumentWithoutStreaming();
>>>>>>> }
>>>>>>> });
>>>>>>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>>>>>> @Override
>>>>>>> public Response call() throws Exceptin {
>>>>>>> return uploadSingleDocumentWithoutStreaming();
>>>>>>> }
>>>>>>> });
>>>>>>> Thread.sleep(15000);
>>>>>>> runConcurrently(CONCURRENT_UPLOADS,new Callable<esponse>() {
>>>>>>> @Override
>>>>>>> public Response call() throws Exception {
>>>>>>> return uploadSingleDocumentWithoutStreaming();
>>>>>>> }
>>>>>>> });
>>>>>>> Thread.sleep(15000);
>>>>>>> runConcurrently(CONCURRENT_UPLOADS,new Callable<Response>() {
>>>>>>> @Override
>>>>>>> public Response call() throws Exception {
>>>>>>> return uploadSingleDocumentWithoutStreaming();
>>>>>>> }
>>>>>>> });
>>>>>>> Thread.sleep(15000);
>>>>>>> runConcurrently(COCURRENT_UPLOADS,new Callable<Response>() {
>>>>>>> @Override
>>>>>>> public Responsecall() throws Exception {
>>>>>>> return uploadSingleDocumentWithoutStreaming();
>>>>>>> }
>>>>>>> });
>>>>>>> Thread.slep(15000);
>>>>>>> .....................................;
>>>>>>> }
>>>>>>>
>>>>>>> This permits to give sometime for the remote host to close the
>>>>>>> connection
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So in the nd it seems that we have troubles in any case when the remote
>>>>>>> host closesthe connection
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Btw I just saw this issue:
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/311
>>>>>>> I checked the code that used AHC 17.7 and what I can see is that the
>>>>>>> connection pool is bypassed even with the property=true
>>>>>>I have in my logs:
>>>>>>> [main] DEBUG
>>>>>>> com.ning.http.client.proviers.grizzly.GrizzlyConnectionsPool:162 -
>>>>>>> [poll] No existing queue for uri [https://dgiposte.orsid.com:443].
>>>>>>>
>>>>>>>
>>>>>>> So, by updating to 1.7.20-SNAPSHOT, the connection pool has been
>>>>>>> magically enabled,and it doesn't seem to work well for me.
>>>>>>>
>>>>>>>
>>>>>>> I'll keep tat pool disabled because it's the behavior of our previous
>>>>>>> applicative version thats near to production and works pretty fine,
>>>>>>> but will open a Github issue because 'm quitting my job next month and
>>>>>>> they want to track when they'll be able to use a connection pool :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/19 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> So it seems I can reproduce this in local.
>>>>>>>
>>>>>>> Disabling the ssl connection pooling solve the problem.
>>>>>>>
>>>>>>>
>>>>>>> We didn't have this problemswith v1.7.7 we previously used.
>>>>>>>
>>>>>>> I just tested with the last commits of 1.7.20-SNAPSHOT, the problem is
>>>>>>> still here.
>>>>>>> The problem doesn't appear on normal AHC requests, neither on multipart
>>>>>>> file upload when I do not use the FeedableBodyGenerator so it seems this
>>>>>>> is the new behavior of FeedableBodyGenerator that has a problem with the
>>>>>>> ssl connection pooling. Will try to investigate this.
>>>>>>>
>>>>>>> By the way, your last commits about selector/worker threads seems ok
>>>>>>> (didn't notice any problem with the ssl pool disabled)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/18 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> Hi
>>>>>>>
>>>>>>> Unfortunatly it seems there's another problem.>>>>>>> On our test environment I often get in the logs:
>>>>>>>
>>>>>>> Caught internal server errorjava.io.IOException: Maximum pooled
>>>>>>> connections exceeded
>>>>>>> at
>>>>>>> comning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHtt
>>>>>>> pClientEvntFilter.cleanup(GrizzlyAsyncHttpProvider.java:1415)
>>>>>>> ~[async-http-client-1.7.20-2846817.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHtt
>>>>>>> pClientEventFilter.onHttpPacketParsed(GrizzlyAsyncHttpProvider.jva:1366
>>>>>>> ) ~[async-http-client-1.7.20-2846817.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.http.HttpCodecFilter.decodeWithTansferEncoding(Ht
>>>>>>> tpCodecFilter.java:1176) ~[grizzly-http-2.3.5.jar:2.35]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpodecFilter.ja
>>>>>>> va:516) ~[grizzly-http-2.3..jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.http.HttpClientFilter.hndleRead(HttpClientFilter.
>>>>>>> java:161) ~[grizzly-http-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolvr$9.execute(ExecutorRes
>>>>>>> olver.java:119) ~[grizzly-framework2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltilterChain.java:288) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filtechain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.jav:136) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.ava:114) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> or.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTr
>>>>>>> anspot.java:546) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(Abstract
>>>>>>> IOStrategy.java:113) ~[grizzly-framewok-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThrea
>>>>>>> dIOStrategy.java:115) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(Wore
>>>>>>> rThreadIOStrategy.java:55) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunn
>>>>>>> able.run(WorkerThreadIOStrategy.java:135)
>>>>>>> ~grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(Abstra
>>>>>>> ctThreadPool.java:565) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractT
>>>>>>> hreadPool.java:545) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> Wrapped by: jva.util.concurrent.ExecutionException:
>>>>>>> java.io.IOException: Maximum pooled connections exceeded
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFuturempl$Sync.innerGet(SafeFutureImpl.j
>>>>>>> ava:359) ~[grizzly-framework-2.3.5.ja:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:265)
>>>>>>> ~[grizzly-framework-2.35.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyResponseFuture.get(Grizzly
>>>>>>> ResponseFuture.java:165) ~[async-http-client-1.7.20-2846817.jar:na]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> boolean canReturnConnection(final Connection c) {
>>>>>>>
>>>>>>> return (DO_NOT_CACHE.get(c) != null ||
>>>>>>> pool.canCacheConnection());
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> private static HttpTransactionContext cleanup(final
>>>>>>> FilterChainContext ctx,
>>>>>> final
>>>>>>> GrizzlyAsyncHttpProvider provider) {
>>>>>>>
>>>>>> final Connection c = ctx.getConection();
>>>>>>> final HttpTransactionContext context =
>>>>>>> provider.getHttpTransactionContext(c);
>>>>>>> context.provider.setHttpTransactionConext(c, null);
>>>>>>> if
>>>>>>> (!context.provider.connectionManager.canReturnConnection(c)) {
>>>>>>> context.abort(new IOException("Maximum pooled
>>>>>>> connections exceeded"));
>>>>>>> } else {
>>>>>>> if
>>>>>>> (!context.provider.connectionManager.returnConnection(context.request,
>>>>>>> c)) {
>>>>>>> ctx.getConnection().close();
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> return context;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We use:
>>>>>>>
>>>>>>> Key=[properties.httpclient.allowSslConnectionPool] Value=[true]
>>>>>>> Key=[properties.httpclient.maximumConnectionsPerHost] Value=[20]
>>>>>>> Key=[properties.httpclient.maximumConnectionsTotal] Value=[30]
>>>>>>> Key=[properties.httpclient.timeout.connection] Value=[10000]
>>>>>>> Key=[properties.httpclient.timeout.request] Value=[30000]
>>>>>>>
>>>>>>>
>>>>>>> In our logs, I can see there are 30 times this log:
>>>>>>>
>>>>>>> Remote closed connection
>>>>>>> (TCPNIOConnection{localSocketAddress={/172.16.104.160:55488
>>>>>>> <http://172.16.104.160:55488> },
>>>>>>> peerSocketAddress={host/172.16.4.100:443}}). Removing from cache
>>>>>>>
>>>>>>> and then it seems no connection is never available and the app is
>>>>>>> blocked.
>>>>>>>
>>>>>>> I'll try tomorrow to disable or increase the pool size, but if you have
>>>>>>> any idea of the problem please let me know.
>>>>>>> We do not run load tests, these are simple functional tests with nearly
>>>>>>> no concurrency.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/17 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> This is good to hear. Thanks for all the feedback and working with us
>>>>>>> to nail this down.
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Thanks,
>>>>>>>
>>>>>>> We already installed the previous snapshot in our nexus because it works
>>>>>>> fine and I'm working on something else now but I'll give you a feedback
>>>>>>> soon to see if this still works fine.
>>>>>>> For us it doesn't really mater which thread is using since we do not use
>>>>>>> Future in our applications for this case.
>>>>>>>
>>>>>>>
>>>>>>> Btw I've been able deploy a main with my code running concurrent
>>>>>>> uploads on a server which has a better connectivity with the remote API
>>>>>>> and it seems I can upload up to 150 concurrent files of 10mb with a heap
>>>>>>> of 250mo in 30 seconds and a throughput near 50mb/s
>>>>>>>
>>>>>>> I don't really know the infrastructure on which my code was deployed but
>>>>>>> I suspect Grizzly is not the bottleneck anymore :)
>>>>>>> When I have more than 150 concurrent uploads the remote endpoint seems
>>>>>>> to timeout under the load.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/16 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I noticed something strange.
>>>>>>> On the FileInputStream I have, I've added a log on the close() of the
>>>>>>> stream which is called once the whole file has been read to be sent to
>>>>>>> the feeder.
>>>>>>>
>>>>>>> 1) If I perform a request (ie my session init request in the previous
>>>>>>> discussions) before doing my multipart upload, the thread that does
>>>>>>> execute the feeding is the thread that fires the request, and not a
>>>>>>> Grizzly worker.
>>>>>>> [Thread-30] INFO Will close file stream java.io.FileInputStream_at_27fe4315
>>>>>>> after having read 1440304 bytes
>>>>>>>
>>>>>>>
>>>>>>> 2) If I don't do any request before firing the multipart upload, the
>>>>>>> thread that does the feeding is a Grizzly threadpool worker thread:
>>>>>>> [Grizzly(22)] INFO Will close file stream
>>>>>>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>>>>>>
>>>>>>>
>>>>>>> Is this normal? I would expect a worker thread to always be used, and
>>>>>>> the main applicative thread that performs the request to never be
>>>>>> blocking. (but it's not such an issue for me, we don't have a reactive
>>>>>> non-blocking app anyway)
>>>>>>>
>>>>>>> It's currently expected behavior. Will need to re-evaluate this based
>>>>>>> on the new semantics of the FeedableBodyGenerator.
>>>>>>> I've committed a change for this. If, upon testing, you find here is
>>>>>>> still an issue, please let us know.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On the 1st case, here's the stacktrae when the feed method is called:
>>>>>>>
>>>>>>> at
>>>>>>> com.ning.http.cient.providers.grizzly.FeedableBodyGenerator.initializeA
>>>>>>> synchronousTransfer(FeedableBodyGenerator.java:178)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsycHttpProvider$BodyGene
>>>>>>> ratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>>>>>>> at
>>>>>>> com.nig.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequ
>>>>>>> est(GrizzlyAsyncHttpProvider.java:564)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHtt
>>>>>>> pClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncttpProvider.java:913)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHtt
>>>>>>> pClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorRes
>>>>>>> olver.java:111)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.rocessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7)
>>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilter
>>>>>>> Chain.java:437)
>>>>>>> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>>>>>>> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>>>>>>> at
>>>>>>> com.ning.http.client.provders.grizzly.GrizzlyAsyncHttpProvider.execute(
>>>>>>> GrizzlyAsyncHttpProvidr.java:307)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.comple
>>>>>> ted(GrizzlyAsyncHttpProvider.java:224)
>>>>>>> at
>>>>>>> com.ning.httpclient.providers.grizzly.GrizzlyAsyncHttpProvider$1.comple
>>>>>>> ted(GrizzlyAsyncHttpProvider.java:210)
>>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Connecti
>>>>>>> onManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvide.java:2289)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(
>>>>>>> GrizzlyAsyncHttpProvider.java:244)
>>>>>>> at
>>>>>>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java
>>>>>>> :534)
>>>>>>>
>>>>>>> So it seems this happen when the handshake has already been done when
>>>>>>> initializeAsynchronousTransfer is called, so that we do not go through
>>>>>>> the HandshakeLisener
>>>>>>>
>>>>>>>
>>>>>>> Notice that this case seems to also happen in a few threads on the 2nd
>>>>>>> case, but most of the threads are Grizzly workers.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/12 Sébastien Lorer <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>>
>>>>>>> Thanks, it seems to work.
>>>>>>>
>>>>>>> I would suggest to throw IOException on the flush method since the feed
>>>>>>> method is supposed to be called here
>>>>>>>
>>>>>>>
>>>>>>> My implementation of flush is:
>>>>>>
>>>>>>> @Override
>>>>>>> public void flush() {
>>>>>>> Part[] partsAray = parts.toArray(new Part[parts.size()]);
>>>>>>> try ( OutputStream utputStream = createFeedingOutputStream() ) {
>>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>>> } catch (Exception e) {
>>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>>> FeedableBodyGenerator",e);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Is this correct? The OutputStream redirects the bytes written to the
>>>>>>> feed(Buffer) method
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> There seems to be some concurrency issue. Because the upload of 1 file
>>>>>>> seems fine, but when using multiple threads, I often get the following
>>>>>>> stack:
>>>>>>> Caused by: java.io.IOException: Stream Closed
>>>>>>> at java.io.FileInputStream.readBytes(Native Method)
>>>>>>> at java.io.FileInputStream.read(FileInputStream.java:242)
>>>>>>> at
>>>>>>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:6
>>>>>>> 2)
>>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>
>>>>>>>
>>>>>>> This is because the flush() method is called multiple times for the same
>>>>>>> request on some cases.
>>>>>>> I guess this is not supposed to happen.
>>>>>>> What I understand is that the flush() method is supposed to be called
>>>>>>> only once.
>>>>>>>
>>>>>>>
>>>>>>> Using debug logging breakpoints I get the following:
>>>>>>>
>>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18 started
>>>>>>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>>>>>>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19 started
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>>>>>>> Adding handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>>>>>>> Completing and removing handshake listener for
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>>>>>>
>>>>>>>
>>>>>>> As you can see the same HandshakeListener seems to be called multiple
>>>>>>> times for the same FeedableBodyGenerator
>>>>>>>
>>>>>>> When this happens, the stack is:
>>>>>>>
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplet
>>>>>>> e(FeedableBodyGenerator.java:198)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseF
>>>>>>> ilter.java:880) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.ja
>>>>>>> va:282) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:27
>>>>>>> 5) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Switchin
>>>>>>> gSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorRes
>>>>>>> olver.java:119) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTr
>>>>>>> ansport.java:546) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(Abstract
>>>>>>> IOStrategy.java:113) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThrea
>>>>>>> dIOStrategy.java:115) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(Worke
>>>>>>> rThreadIOStrategy.java:55) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunn
>>>>>>> able.run(WorkerThreadIOStrategy.java:135)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(Abstra
>>>>>>> ctThreadPool.java:565) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractT
>>>>>>> hreadPool.java:545) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So I tried with the following code:
>>>>>>>
>>>>>>> private boolean alreadyFlushed = false;
>>>>>>> @Override
>>>>>>> public synchronized void flush() {
>>>>>>> if ( alreadyFlushed ) {
>>>>>>> return;
>>>>>>> }
>>>>>>> startFeeding();
>>>>>>> alreadyFlushed = true;
>>>>>>> }
>>>>>>>
>>>>>>> It works fine when upload small files.
>>>>>>>
>>>>>>> But with larger files, I often get TimeoutException stacks for some of
>>>>>>> the threads:
>>>>>>>
>>>>>>> Caused by: java.util.concurrent.TimeoutException
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.j
>>>>>>> ava:367)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.
>>>>>>> block(FeedableBodyGenerator.java:349)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.
>>>>>>> blockUntilQueueFree(FeedableBodyGenerator.java:339)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.
>>>>>>> feed(FeedableBodyGenerator.java:306)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Did I miss something?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Committed another small change. Please make sure you're at the latest
>>>>>>> when you build.
>>>>>>>
>>>>>>> -rl
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>> Okay, I've committed another set of refactorings to the
>>>>>>> FeedableBodyGenerator.
>>>>>>>
>>>>>>> For your use case, you should extend FeedableBodyGenerator.SimpleFeeder.
>>>>>>>
>>>>>>> Let me know if you run into issues.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Yes I think it would, so that I can feed the queue at once
>>>>>>>
>>>>>>> One thread will be locked during the feeding for nothing but it's not a
>>>>>>> real problem in my usecase.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Would having a different listener that will be notified once async
>>>>>>> transferring has been started work better for you?
>>>>>>>
>>>>>>> Something like:
>>>>>>>
>>>>>>> onAsyncTransferInitiated() {
>>>>>>> // invoke your feed method
>>>>>>> }
>>>>>>>
>>>>>>> ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Unfortunatly I won't be able to use the Feeder non-blocking stuff for
>>>>>>> now, because of how the multipart request in handled in AHC
>>>>>>>
>>>>>>>
>>>>>>> Here's my feeding method:
>>>>>>>
>>>>>>> public void feed() throws IOException {
>>>>>>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>>>>>>> try ( OutputStream outputStream = createFeedingOutputStream() ) {
>>>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>>>> } catch (Exception e) {
>>>>>>> throw new IllegalStateException("Unable to feed the
>>>>>>> FeedableBodyGenerator",e);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> As you can see, the multipart Parts array can only be pushed to the
>>>>>>> OutputStream, I don't have any way to "pull" the data when the canFeed()
>>>>>>> method is triggered.
>>>>>>>
>>>>>>>
>>>>>>> But I've seen that there's a com.ning.http.multipart.MultipartBody#read
>>>>>>> that seems to provide a memory efficient way to pull data from a
>>>>>>> Multipart body...
>>>>>>>
>>>>>>> Should see what I come up with this
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> It seems the Feeder is highly recommended but not mandatory so I tried
>>>>>>> without.
>>>>>>>
>>>>>>> With my existing code it seems there is a synchronization problem.
>>>>>>>
>>>>>>>
>>>>>>> The feeding threads get locked to the prematureFeed.get();
>>>>>>>
>>>>>>> So the Grizzly kernel threads are unable to acquire the lock required to
>>>>>>> enter the initializeAsynchronousTransfer method
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Will try with an implementation of Feeder
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> Hmmm it seems I have a problem with one of your maven plugins. I'll try
>>>>>>> to bypass it, but for info:
>>>>>>>
>>>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>>>> [WARNING]
>>>>>>> [WARNING] Some problems were encountered while building the effective
>>>>>>> settings
>>>>>>> [WARNING] 'profiles.profile[default].repositories.repository.id
>>>>>>> <http://repositories.repository.id> ' must be unique but found duplicate
>>>>>>> repository with id fullsix-maven-repository @
>>>>>>> /home/slorber/.m2/settings.xml
>>>>>>> [WARNING]
>>>>>>> [INFO] Scanning for projects...
>>>>>>> [INFO]
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO]
>>>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>>>>>>> async-http-client ---
>>>>>>> [INFO]
>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven) @
>>>>>>> async-http-client ---
>>>>>>> [INFO]
>>>>>>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-versions) @
>>>>>>> async-http-client ---
>>>>>>> [INFO]
>>>>>>> [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
>>>>>>> async-http-client ---
>>>>>>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>>>> [INFO]
>>>>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
>>>>>>> async-http-client ---
>>>>>>> [INFO] Compiling 158 source files to
>>>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>>>> [INFO]
>>>>>>> [INFO] --- animal-sniffer-maven-plugin:1.6:check (check-java-1.5-compat)
>>>>>>> @ async-http-client ---
>>>>>>> [INFO] Checking unresolved references to
>>>>>>> org.codehaus.mojo.signature:java15:1.0
>>>>>>> [ERROR] Undefined reference:
>>>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>>>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/
>>>>>>> grizzly/FeedableBodyGenerator.class
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 8.747s
>>>>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>>>>> [INFO] Final Memory: 30M/453M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>>>> (check-java-1.5-compat) on project async-http-client: Signature errors
>>>>>>> found. Verify them and put @IgnoreJRERequirement on them. -> [Help 1]
>>>>>>> [ERROR]
>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>>>>>>> -e switch.
>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>>>>>> [ERROR]
>>>>>>> [ERROR] For more information about the errors and possible solutions,
>>>>>>> please read the following articles:
>>>>>>> [ERROR] [Help 1]
>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> Ok thank you, I'll try to implement that today and will give you my
>>>>>>> feedback :)
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Okay,
>>>>>>>
>>>>>>> I've committed my initial changes to the AHC repository. Here's a
>>>>>>> summary of the changes:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Improvements to the FeedableBodyGenerator (Grizzly's).
>>>>>>> - Don't allow queueing of data before initiateAsyncTransfer has been
>>>>>>> invoked. In low memory
>>>>>>> heaps, this could lead to an OOM if the source is feeding too fast. The
>>>>>>> new behavior is to
>>>>>>> block until initiateAsyncTransfer is called, at which time the blocked
>>>>>>> thread may proceed with
>>>>>>> the feed operation.
>>>>>>> - Introduce the concept of a Feeder. Implementations are responsible,
>>>>>>> at a high level, for:
>>>>>>> + letting the provider know that data is available to be fed without
>>>>>>> blocking
>>>>>>> + allowing the registration of a callback that the Feeder
>>>>>>> implementation may invoke
>>>>>>> to signal that more data is available, if it wasn't available at a
>>>>>>> previous point in time.
>>>>>>> - When using a Feeder with a secure request, the SSL handshake will be
>>>>>>> kicked off by the
>>>>>>> initiateAsyncTransfer call, but feeding of data will not occur until
>>>>>>> the handshake is complete.
>>>>>>> This is necessary as the SSLFilter will queue up all writes until the
>>>>>>> handshake is complete,
>>>>>>> and currently, the buffer isn't tied in with the transport flow control
>>>>>>> mechanism.
>>>>>>> NOTE: This new SSL behavior is not currently applied when invoking the
>>>>>>> feed() method
>>>>>>> outside the context of a Feeder. Still need to address that.
>>>>>>> - Exposed configuration of the async write queue limit through the
>>>>>>> FeedableBodyGenerator.
>>>>>>> This is an improvement on using a TransportCustomizer as any
>>>>>>> configuration there is
>>>>>>> transport-wide, and therefor applied to all Connections. By exposing
>>>>>>> it here, each feeder
>>>>>>> may have a different byte limit.
>>>>>>> - Improved documentation for this class
>>>>>>>
>>>>>>> I recommend reading through the javadoc comments in the source [1] for
>>>>>>> FeedableBodyGenerator (comments welcome).
>>>>>>> Additionally, I would re-work your code to leverage the Feeder instead
>>>>>>> of calling feed() directly.
>>>>>>>
>>>>>>> If you have issues implementing Feeder, do let us know.
>>>>>>>
>>>>>>> If you have additional questions, again, let us know.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -rl
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/
>>>>>>> main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.j
>>>>>>> ava
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> So in the end I've end up with an implementation that's working for me.
>>>>>>>
>>>>>>>
>>>>>>> I think there are 2 bugs:
>>>>>>>
>>>>>>> 1) The bytes can accumulate in the FeedableBodyGenerator queue if the
>>>>>>> initialize(ctx) method is not called fast enough.
>>>>>>> This can be solved by using a BlockingQueue of size 1 and the put()
>>>>>>> method.
>>>>>>>
>>>>>>> 2) Once the context is injected, the FeedableBodyGenerator flushes the
>>>>>>> queue.
>>>>>>> The matter is that if the connection is new, not warmed up by a
>>>>>>> previous request, then the SSL handshake is not done yet, and it seems
>>>>>>> that the bytes are accumulated in some part of the SSL filter which
>>>>>>> doesn't deliver them to the connection until the handshake has
>>>>>>> completed, so c.canWrite() continues to return true.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have replaced some part of the FeedableBodyGenerator to test this and
>>>>>>> it works pretty fine. See what I have changed:
>>>>>>>
>>>>>>> 1)
>>>>>>> private final BlockingQueue<BodyPart> queue = new
>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>
>>>>>>>
>>>>>>> 2)
>>>>>>> public void feed(final Buffer buffer, final boolean last) throws
>>>>>>> IOException {
>>>>>>> try {
>>>>>>> queue.put(new BodyPart(buffer, last));
>>>>>>> } catch (InterruptedException e) {
>>>>>>> throw new RuntimeException(e);
>>>>>>> }
>>>>>>> queueSize.incrementAndGet();
>>>>>>>
>>>>>>> if (context != null) {
>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private void blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>>>>> fcc) {
>>>>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>>>>> try { Thread.sleep(10); } catch ( Exception e ) { throw new
>>>>>>> RuntimeException(e); }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private boolean connectionIsReadyToWrite(FilterChainContext fcc) {
>>>>>>> Connection connection = fcc.getConnection();
>>>>>>> SSLEngine sslEngine = SSLUtils.getSSLEngine(connection);
>>>>>>> return sslEngine != null && !SSLUtils.isHandshaking(sslEngine);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>>
>>>>>>> We had come to similar conclusions on this end. I'm still working
>>>>>>> through testing the idea I mentioned previously (took longer than I
>>>>>>> expected - sorry).
>>>>>>> I hope to have something for you to test very soon.
>>>>>>>
>>>>>>> Note that it will be taking the above into account as well.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>> I have tried to put a while ( context == null ) Thread.sleep but it
>>>>>>> doesn't seem to work, when the context gets injected, after the sleeps,
>>>>>>> there's an OOM
>>>>>>>
>>>>>>> So I hope you'll have more success with your alternative :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have done another test, remember my code that worked, which previously
>>>>>>> "warmed" the Thread with an useless request.
>>>>>>>
>>>>>>> private Runnable uploadSingleDocumentRunnable = new Runnable() {
>>>>>>> @Override
>>>>>>> public void run() {
>>>>>>> try {
>>>>>>> getUselessSessionCode();
>>>>>>> Thread.sleep(X);
>>>>>>> uploadSingleDocument();
>>>>>>> } catch ( Exception e ) {
>>>>>>> throw new RuntimeException("file upload failed",e);
>>>>>>> }
>>>>>>> }
>>>>>>> };
>>>>>>>
>>>>>>> I have put a sleep of X between the useless warmup request, and the real
>>>>>>> upload request
>>>>>>>
>>>>>>>
>>>>>>> What I noticed is that there is a very different behavior according to
>>>>>>> the value of X
>>>>>>>
>>>>>>>
>>>>>>> Under 10 seconds, it seems the stuff is still warm, I can upload the
>>>>>>> documents.
>>>>>>> Around 10 seconds I get a stack which seems to be "connection closed" or
>>>>>>> something
>>>>>>> Above 10 seconds, I get OOM like if the stuff wasn't warm.
>>>>>>>
>>>>>>>
>>>>>>> The stacks I get for 10 seconds looks like
>>>>>>>
>>>>>>> Caused by: javax.net.ssl.SSLException: SSLEngine is CLOSED
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext
>>>>>>> .java:295) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionCont
>>>>>>> ext.java:238) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:3
>>>>>>> 20) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Switchin
>>>>>>> gSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorRes
>>>>>>> olver.java:111) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainCo
>>>>>>> ntext.java:853) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainCo
>>>>>>> ntext.java:720) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(
>>>>>>> FeedableBodyGenerator.java:133)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(Feedab
>>>>>>> leBodyGenerator.java:95) ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>
>>>>>>> I think I got some other different stacks saying Connection Closed
>>>>>>> Remotely or something like that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So it seems that something is bound to my thread, and it stays bound to
>>>>>>> it for about 10 seconds, do you have any idea what it could be?
>>>>>>> (My connection timeout setting seems to have no effect on this 10s
>>>>>>> threshold)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> That is one solution. I'm working out an alternative right now. Stay
>>>>>>> tuned!
>>>>>>>
>>>>>>>
>>>>>>> Anyway it's not a problem, I think the FeedableBodyGenerator.feed()
>>>>>>> method just has to block until a context has been (and ThreadCache
>>>>>>> initialized) to avoid OOM errors
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>> What is very strange is that I tested with/without the same sessionCode
>>>>>>> with our previous code, the one not using FeedableBodyGenerator, which
>>>>>>> has a high memory consumption.
>>>>>>> Despites the fact it had high memory consumption, it seems work fine to
>>>>>>> upload multiple documents if allocated with a large heap, and the
>>>>>>> sessionCode seems to have no effect.
>>>>>>>
>>>>>>> On the new impl using the FeedableBodyGenerator, the sessionCode sent as
>>>>>>> a multipart bodypart seems to have an effect.
>>>>>>>
>>>>>>> I have tried to feed the queue before sending the request to AHC, but
>>>>>>> this leads to this exception (with/without sessionCode switching)
>>>>>>> Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(
>>>>>>> GrizzlyAsyncHttpProvider.java:528)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTime
>>>>>>> out(GrizzlyAsyncHttpProvider.java:361)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleT
>>>>>>> imeoutFilter.java:383)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleT
>>>>>>> imeoutFilter.java:362)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedE
>>>>>>> xecutor.java:158)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>>>>> a:1110)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>>>>> va:603)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> By the way, by using a low timeout with the same sessioncode, I got the
>>>>>>> following NPE:
>>>>>>>
>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(Feeda
>>>>>>> bleBodyGenerator.java:184)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQ
>>>>>>> ueueFree(FeedableBodyGenerator.java:167)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(
>>>>>>> FeedableBodyGenerator.java:124)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(Feedab
>>>>>>> leBodyGenerator.java:94)
>>>>>>>
>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext httpCtx =
>>>>>>> getHttpTransactionContext(c);
>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>
>>>>>>> I guess the httpCtx is not already available to be aborted
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>
>>>>>>> This is right, here's a log I have when I use the same session code, ie
>>>>>>> the remote host is blocking the data or something.
>>>>>>> This is obtained by running 5 parallel uploads.
>>>>>>>
>>>>>>> Flushing queue of size 0 with allowBlocking = false
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 97 with allowBlocking = false
>>>>>>> Flushing queue of size 100 with allowBlocking = false
>>>>>>> Flushing queue of size 160 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 0 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>> Dumping heap to /ome/lorber/ureau/om ...
>>>>>>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Disconnected from the target VM, address: '127.0.0.1:49268
>>>>>>> <http://127.0.0.1:49268> ', transport: 'socket'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Otherwise, with different session codes, I get the following:
>>>>>>>
>>>>>>> Flushing queue of size 0 with allowBlocking = false
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 0 with allowBlocking = false
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 0 with allowBlocking = false
>>>>>>> Flushing queue of size 0 with allowBlocking = false
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> Flushing queue of size 1 with allowBlocking = true
>>>>>>> ... and this continues without OOM
>>>>>>>
>>>>>>>
>>>>>>> So, this seems to be the problem.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think it would be great to be able to be able to choose the queue impl
>>>>>>> behind that FeedableBodyGenerator, like I suggested in my pull request.
>>>>>>>
>>>>>>> See here:
>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b05
>>>>>>> 5bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/
>>>>>>> FeedableBodyGenerator.java
>>>>>>>
>>>>>>> Using a LinkedBlockingQueue seems to be a nice idea in this context, and
>>>>>>> in my case I would probably use a queue of size 1
>>>>>>>
>>>>>>>
>>>>>>> This would handle the blocking of the feed method, without having to
>>>>>>> use this:
>>>>>>> if (context != null) {
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Or perhaps the feed() method have to wait until a context is set in the
>>>>>>> BodyGenerator ?
>>>>>>>
>>>>>>>
>>>>>>> I think it would be more clear if the initializeAsynchronousTransfer
>>>>>>> simply didn't flush the queue but just setup the context.
>>>>>>> Then the feed method would block until there's a context set, and then
>>>>>>> flush the queue with blocking behavior.
>>>>>>>
>>>>>>> This is probably the next step, but as we are using AHC for async, it
>>>>>>> would probably be great if that blocking feed() method was called in a
>>>>>>> worker thread instead of our main thread.
>>>>>>> I won't use this but someone who really wants a non-blocking impl of
>>>>>>> performant multipart fileupload would probably need this, or will use an
>>>>>>> ExecutorService for the feeding operations as a workaround.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks again for your reactivity
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>>
>>>>>>> I've integrated this change and it works fine except a little detail.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I'm uploading files to a third party API (a bit like S3).
>>>>>>> This API requires a "sessionCode" in each request. So there is a
>>>>>>> multipart StringPart with that SessionCode.
>>>>>>>
>>>>>>> We used to have a cache which holds the sessionCode 30min per user so
>>>>>>> that we do not need to init a new session each time.
>>>>>>>
>>>>>>> I had troubles in this very specific case: when I upload 5 docs with the
>>>>>>> same session code.
>>>>>>> When I remove the cache and use 5 different session codes, it works
>>>>>>> fine.
>>>>>>>
>>>>>>> I guess the remote service is blocking concurrent uploads with the same
>>>>>>> session code. I don't know at all.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Where I want to go is that I wouldn't have expected Grizzly to OOM
>>>>>>>
>>>>>>>
>>>>>>> Avertissement: Exception during FilterChain execution
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:34
>>>>>>> 2)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConne
>>>>>>> ctionContext.java:392)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext
>>>>>>> .java:272)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionCont
>>>>>>> ext.java:238)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:3
>>>>>>> 20)
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Switchin
>>>>>>> gSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorRes
>>>>>>> olver.java:111)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainCo
>>>>>>> ntext.java:196)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainC
>>>>>>> ontext.java:220)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFil
>>>>>>> ter.java:383)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.ja
>>>>>>> va:278)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:27
>>>>>>> 5)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Switchin
>>>>>>> gSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorRes
>>>>>>> olver.java:119)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTr
>>>>>>> ansport.java:546)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(Abstract
>>>>>>> IOStrategy.java:113)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Caused by: java.util.concurrent.TimeoutException: null
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.j
>>>>>>> ava:367) ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(Feeda
>>>>>>> bleBodyGenerator.java:177) ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQ
>>>>>>> ueueFree(FeedableBodyGenerator.java:167)
>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(
>>>>>>> FeedableBodyGenerator.java:124)
>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(Feedab
>>>>>>> leBodyGenerator.java:94) ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> multipart.body.generator.feeder.buffer=100000 -> size of each Buffer
>>>>>>> sent to the FeedableBodyGenerator
>>>>>>> transport.max.pending.bytes=1000000
>>>>>>> (I tried other settings, including AUTO_SIZE)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Do you have any idea why is there an OOM with these settings?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Perhaps it is because the feed() method of FeedableBodyGenerator doesn't
>>>>>>> block until the context is initialized.
>>>>>>> I guess the initializeAsynchronousTransfer is called only once the
>>>>>>> connection is established, and perhaps a lot of Buffer are added to the
>>>>>>> queue...
>>>>>>> Yes, it's invoked once the request has been dispatched, so if the
>>>>>>> generator is fed a lot before the request, I could see this happening.
>>>>>>> I'll see what I can do to alleviate that case.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> But I'm not sure at all because the session code is transmitted as a
>>>>>>> BodyPart and I get the same problem if i put it as the first or last
>>>>>>> multipart.
>>>>>>>
>>>>>>> It's not a big deal, perhaps I should always use a different session
>>>>>>> code for concurrent operations but I'd like to be sure that we won't
>>>>>>> have this issue in production...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Good catch. Fixed.
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> There's a little mistake in the grizzly ahc provider relative to the
>>>>>>> write queue size.
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe141
>>>>>>> 13ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/
>>>>>>> grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>
>>>>>>> As you can see, the TransportCustomizer is called, and then the write
>>>>>>> queue size (among other things) is set to AUTO_SIZE (instead of
>>>>>>> previously UNLIMITED)
>>>>>>>
>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>
>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>
>>>>>>>
>>>>>>> I think the default settings like this AUTO_SIZE attribute should be set
>>>>>>> before the customization of the transport, or they would override the
>>>>>>> value we customized.
>>>>>>>
>>>>>>>
>>>>>>> This is actually my case, since I can't reproduce my "bug" which is
>>>>>>> "high memory consumption", even when using -1 / UNLIMITED in the
>>>>>>> TransportCustomizer.
>>>>>>>
>>>>>>>
>>>>>>> This could work fine for me with AUTO_SIZE, but I'd rather be able to
>>>>>>> tune this parameter during load tests to see the effect.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> Thanks i will ckeck that on monday. I can now upload a 500m file with
>>>>>>> 40m heap size ;)
>>>>>>>
>>>>>>> Envoyé de mon iPhone
>>>>>>>
>>>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <ryan.lubke_at_oracle.com> a écrit :
>>>>>>>
>>>>>>> I'm going to be updating the Grizzly provider such that AUTO_SIZE (not
>>>>>>> AUTO_TUNE) is the default, so you can avoid the use of the
>>>>>>> TransportCustomizer.
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>> Regarding your tuning question, I would probably set the value to
>>>>>>> AsyncQueueWriter.AUTO_TUNE (this will be four times the socket write
>>>>>>> buffer) and see how that works.
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>> A question first. With these changes, your memory usage is more inline
>>>>>>> with what you were looking for?
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> By the way, do you have any idea when the 1.7.20 will be released (with
>>>>>>> these new improvements?)
>>>>>>>
>>>>>>> We would like to know if we wait for a release or if we install our own
>>>>>>> temp release on Nexus :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>> Thank you, it works fine!
>>>>>>>
>>>>>>>
>>>>>>> I just had to modify a single line after your commit.
>>>>>>>
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initiali
>>>>>>> zeTransport
>>>>>>> ->
>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConne
>>>>>>> ction(10000);
>>>>>>>
>>>>>>>
>>>>>>> If I let the initial value (-1) it won't block, canWrite always returns
>>>>>>> true
>>>>>>>
>>>>>>>
>>>>>>> Btw, on AHC I didn't find any way to pass this value as a config
>>>>>>> attribute, neither the size of the write buffer you talked about.
>>>>>>>
>>>>>>> So in the end, is there a way with current AHC code to use this
>>>>>>> "canWrite = false" behavior?
>>>>>>> If not, can you please provide a way to set this behavior on v1.7.20 ?
>>>>>>> thanks
>>>>>>>
>>>>>>>
>>>>>>> PS: does it make sens to use the same number of bytes un the
>>>>>>> feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000); ?
>>>>>>> do you have any tuning recommandation?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Please disregard.
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>> Sébastien,
>>>>>>>
>>>>>>> Could you also provide a sample of how you're performing your feed?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -rl
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>> Sébastien,
>>>>>>>
>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>
>>>>>>> By default, Grizzly will configure the async write queue length to be
>>>>>>> four times the write buffer size (which is based off the socket write
>>>>>>> buffer).
>>>>>>> When this queue exceeds this value, canWrite() will return false.
>>>>>>>
>>>>>>> When this occurs, you can register a WriteListener to be notified when
>>>>>>> the queue length is below the configured max and then simulate blocking
>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>>
>>>>>>> final FutureImpl<Boolean> future = Futures.createSafeFuture();
>>>>>>>
>>>>>>> // Connection may be obtained by calling
>>>>>>> FilterChainContext.getConnection().
>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>
>>>>>>> @Override
>>>>>>> public void onWritePossible() throws Exception {
>>>>>>> future.result(Boolean.TRUE);
>>>>>>> }
>>>>>>>
>>>>>>> @Override
>>>>>>> public void onError(Throwable t) {
>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>> }
>>>>>>> });
>>>>>>>
>>>>>>> try {
>>>>>>> final long writeTimeout = 30;
>>>>>>> future.get(writeTimeout, TimeUnit.MILLISECONDS);
>>>>>>> } catch (ExecutionException e) {
>>>>>>> HttpTransactionContext httpCtx =
>>>>>>> HttpTransactionContext.get(connection);
>>>>>>> httpCtx.abort(e.getCause());
>>>>>>> } catch (Exception e) {
>>>>>>> HttpTransactionContext httpCtx =
>>>>>>> HttpTransactionContext.get(connection);
>>>>>>> httpCtx.abort(e);
>>>>>>> }
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> [1]
>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSin
>>>>>>> k.html.
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Ryan, I've did some other tests.
>>>>>>>
>>>>>>>
>>>>>>> It seems that using a blocking queue in the FeedableBodyGenerator is
>>>>>>> totally useless because the thread consuming it is not blocking and the
>>>>>>> queue never blocks the feeding, which was my intention in the beginning.
>>>>>>> Maybe it depends on the IO strategy used?
>>>>>>> I use AHC default which seems to use SameThreadIOStrategy so I don't
>>>>>>> think it's related to the IO strategy.
>>>>>>>
>>>>>>>
>>>>>>> So in the end I can upload a 70m file with a heap of 50m, but I have to
>>>>>>> put a Thread.sleep(30) between each 100k Buffer send to the
>>>>>>> FeedableBodyGenerator
>>>>>>>
>>>>>>> The connection with the server is not good here, but in production it is
>>>>>>> normally a lot better as far as I know.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I've tried things like
>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConne
>>>>>>> ction(100000); but it doesn't seem to work for me.
>>>>>>>
>>>>>>> I'd like the Grizzly internals to block when there are too much pending
>>>>>>> bytes to send. Is it possible?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> PS: I've just been able to send a 500mo file with 100mo heap, but it
>>>>>>> needed a sleep of 100ms between each 100k Buffer sent to the
>>>>>>> bodygenerator
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>
>>>>>>> By chance do you if I can remove the MessageCloner used in the SSL
>>>>>>> filter?
>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>
>>>>>>> It seems to allocate a lot of memory.
>>>>>>> I don't really understand why messages have to be cloned, can I remove
>>>>>>> this? How?
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>> I'm trying to send a 500m file for my tests with a heap of 400m.
>>>>>>>
>>>>>>> In our real use cases we would probably have files under 20mo but we
>>>>>>> want to reduce the memory consumption because we can have x parallel
>>>>>>> uploads on the same server according to the user activity.
>>>>>>>
>>>>>>> I'll try to check if using this BodyGenerator reduced the memory
>>>>>>> footprint or if it's almost like before.
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> At this point in time, as far as the SSL buffer allocation is concerned,
>>>>>>> it's untunable.
>>>>>>>
>>>>>>> That said, feel free to open a feature request.
>>>>>>>
>>>>>>> As to your second question, there is no suggested size. This is all
>>>>>>> very application specific.
>>>>>>>
>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> I have seen a lot of buffers which have a size of 33842 and it seems the
>>>>>>> limit is near half the capacity.
>>>>>>>
>>>>>>> Perhaps there's a way to tune that buffer size so that it consumes less
>>>>>>> memory?
>>>>>>> Is there an ideal Buffer size to send to the feed method?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>
>>>>>>> Regarding the OOM: as it stands now, for each new buffer that is passed
>>>>>>> to the SSLFilter, we allocate a buffer twice the size in order to
>>>>>>> accommodate the encrypted result. So there's an increase.
>>>>>>>
>>>>>>> Depending on the socket configurations of both endpoints, and how fast
>>>>>>> the remote is reading data, it could
>>>>>>> be the write queue is becoming too large. We do have a way to detect
>>>>>>> this situation, but I'm pretty sure
>>>>>>> the Grizzly internals are currently shielded here. I will see what I
>>>>>>> can do to allow users to leverage this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I've made my pull request.
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>
>>>>>>> With my usecase it works, the file is uploaded like before.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>
>>>>>>> Is it possible that SSL doesn't allow to stream the body or something
>>>>>>> like that?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> In memory, I have a lot of:
>>>>>>> - HeapByteBuffer
>>>>>>> Which are hold by SSLUtils$3
>>>>>>> Which are hold by BufferBuffers
>>>>>>> Which are hold by WriteResult
>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>
>>>>>>>
>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:34
>>>>>>> 2)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConne
>>>>>>> ctionContext.java:392)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext
>>>>>>> .java:272)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionCont
>>>>>>> ext.java:227)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:3
>>>>>>> 19)
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$Switchin
>>>>>>> gSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorRes
>>>>>>> olver.java:111)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(Defau
>>>>>>> ltFilterChain.java:288)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(De
>>>>>>> faultFilterChain.java:206)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilt
>>>>>>> erChain.java:136)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilt
>>>>>>> erChain.java:114)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:7
>>>>>>> 7)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainCo
>>>>>>> ntext.java:853)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainCo
>>>>>>> ntext.java:720)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(
>>>>>>> FeedableBodyGenerator.java:132)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(Feedab
>>>>>>> leBodyGenerator.java:101)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$Feed
>>>>>>> BodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>> at
>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed
>>>>>>> (MultipartBodyGeneratorFeeder.java:144)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any idea?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Ryan thanks, it works fine, I'll make a pull request on AHC tomorrow
>>>>>>> with a better code using the same Part classes that already exist.
>>>>>>>
>>>>>>> I created an OutputStream that redirects to the BodyGenerator feeder.
>>>>>>>
>>>>>>> The problem I currently have is that the feeder feeds the queue faster
>>>>>>> than the async thread polling it :)
>>>>>>> I need to expose a limit to that queue size or something, will work on
>>>>>>> that, it will be better than a thread sleep to slow down the filepart
>>>>>>> reading
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>> Yes, something like that. I was going to tackle adding something like
>>>>>>> this today. I'll follow up with something you can test out.
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Ok thanks!
>>>>>>>
>>>>>>> I think I see what I could do, probably something like that:
>>>>>>>
>>>>>>>
>>>>>>> FeedableBodyGenerator bodyGenerator = new FeedableBodyGenerator();
>>>>>>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder = new
>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>> Request uploadRequest1 = new RequestBuilder("POST")
>>>>>>> .setUrl("url")
>>>>>>> .setBody(bodyGenerator)
>>>>>>> .build();
>>>>>>>
>>>>>>> ListenableFuture<Response> asyncRes = asyncHttpClient
>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>> .execute(new AsyncCompletionHandlerBase());
>>>>>>>
>>>>>>>
>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>
>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>
>>>>>>>
>>>>>>> Does it seem ok to you?
>>>>>>>
>>>>>>> I guess it could be interesting to provide that
>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>> people may want to achieve the same thing
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I would like to know if it's possible to upload a file with AHC /
>>>>>>> Grizzly in streaming, I mean without loading the whole file bytes in
>>>>>>> memory.
>>>>>>>
>>>>>>> The default behavior seems to allocate a byte[] which contans the whole
>>>>>>> file, so it means that my server can be OOM if too many users upload a
>>>>>>> large file in the same time.
>>>>>>>
>>>>>>>
>>>>>>> I've tryied with a Heap and ByteBuffer memory managers, with
>>>>>>> reallocate=true/false but no more success.
>>>>>>>
>>>>>>> It seems the whole file content is appended wto the BufferOutputStream,
>>>>>>> and then the underlying buffer is written.
>>>>>>>
>>>>>>> At least this seems to be the case with AHC integration:
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546
>>>>>>> 110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttp
>>>>>>> client/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>
>>>>>>>
>>>>>>> So, is there a way to patch AHC to stream the file so that I could
>>>>>>> eventually consume only 20mo of heap while uploading a 500mo file?
>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>> I didn't notice anything related to that in the documentation.
>>>>>>> It's possible with the FeedableBodyGenerator. But if you're tied to
>>>>>>> using Multipart uploads, you'd have to convert the multipart data to
>>>>>>> Buffers manually and send using the FeedableBodyGenerator.
>>>>>>> I'll take a closer look to see if this area can be improved.
>>>>>>>
>>>>>>>
>>>>>>> Btw in my case it is a file upload. I receive a file with CXF and have
>>>>>>> to transmit it to a storage server (like S3). CXF doesn't consume memory
>>>>>>> bevause it is streaming the large fle uploads to the file system, and
>>>>>>> then provides an input stream on that file.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>>>