From: Ryan Lubke <ryan.lubke_at_oracle.com>
Date: Mon, 16 Sep 2013 11:26:43 -0700

Ryan Lubke wrote:
>
>
> Sébastien Lorber wrote:
>>
>>
>>
>> I noticed something strange.
>> On the FileInputStream I have, I've added a log on the close() of the
>> stream which is called once the whole file has been read to be sent
>> to the feeder.
>>
>> 1) If I perform a request (ie my session init request in the previous
>> discussions) before doing my multipart upload, the thread that does
>> execute the feeding is the thread that fires the request, and not a
>> Grizzly worker.
>> [Thread-30] INFO Will close file stream
>> java.io.FileInputStream_at_27fe4315 after having read 1440304 bytes
>>
>>
>> 2) If I don't do any request before firing the multipart upload, the
>> thread that does the feeding is a Grizzly threadpool worker thread:
>> [Grizzly(22)] INFO Will close file stream
>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>
>>
>> Is this normal? I would expect a worker thread to always be used, and
>> the main applicative thread that performs the request to never be
>> blocking. (but it's not such an issue for me, we don't have a
>> reactive non-blocking app anyway)
> It's currently expected behavior. Will need to re-evaluate this
> based on the new semantics of the FeedableBodyGenerator.
I've committed a change for this. If, upon testing, you find there is
still an issue, please let us know.
>>
>>
>> On the 1st case, here's the stacktrace when the feed method is called:
>>
>> *at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)*
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>> at
>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>> at
>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
>> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
>> *at
>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)*
>>
>> So it seems this happen when the handshake has already been done when
>> *initializeAsynchronousTransfer* is called, so that we do not go
>> through the HandshakeListener
>>
>>
>> Notice that this case seems to also happen in a few threads on the
>> 2nd case, but most of the threads are Grizzly workers.
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com
>> <mailto:lorber.sebastien_at_gmail.com>>
>>
>>
>>
>> Hi.
>>
>>
>> Thanks, it seems to work.
>>
>> I would suggest to throw IOException on the flush method since
>> the feed method is supposed to be called here
>>
>>
>> My implementation of flush is:
>>
>> @Override
>> public void flush() {
>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>> try ( OutputStream outputStream =
>> createFeedingOutputStream() ) {
>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>> } catch (Exception e) {
>> throw new IllegalStateException("Unable to feed the
>> FeedableBodyGenerator",e);
>> }
>> }
>>
>> Is this correct? The OutputStream redirects the bytes written to
>> the feed(Buffer) method
>>
>>
>>
>>
>>
>>
>> There seems to be some concurrency issue. Because the upload of 1
>> file seems fine, but when using multiple threads, I often get the
>> following stack:
>> Caused by: java.io.IOException: Stream Closed
>> at java.io.FileInputStream.readBytes(Native Method)
>> at java.io.FileInputStream.read(FileInputStream.java:242)
>> at
>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>> at com.ning.http.multipart.Part.send(Part.java:331)
>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>
>>
>> This is because the flush() method is called multiple times for
>> the same request on some cases.
>> I guess this is not supposed to happen.
>> What I understand is that the flush() method is supposed to be
>> called only once.
>>
>>
>> Using debug logging breakpoints I get the following:
>>
>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0
>> started
>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1
>> started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2
>> started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3
>> started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4
>> started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5
>> started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6
>> started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7
>> started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8
>> started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9
>> started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10
>> started
>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11
>> started
>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12
>> started
>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13
>> started
>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14
>> started
>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15
>> started
>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16
>> started
>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17
>> started
>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18
>> started
>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19
>> started
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>
>>
>> As you can see the same HandshakeListener seems to be called
>> multiple times for the same FeedableBodyGenerator
>>
>> When this happens, the stack is:
>>
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>> at
>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>> at
>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>
>>
>>
>>
>> So I tried with the following code:
>>
>> private boolean alreadyFlushed = false;
>> @Override
>> public synchronized void flush() {
>> if ( alreadyFlushed ) {
>> return;
>> }
>> startFeeding();
>> alreadyFlushed = true;
>> }
>>
>> It works fine when upload small files.
>>
>> But with larger files, I often get TimeoutException stacks for
>> some of the threads:
>>
>> Caused by: java.util.concurrent.TimeoutException
>> at
>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>> at
>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>>
>>
>>
>>
>> Did I miss something?
>>
>>
>>
>>
>>
>>
>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com
>> <mailto:ryan.lubke_at_oracle.com>>
>>
>> Committed another small change. Please make sure you're at
>> the latest when you build.
>>
>> -rl
>>
>>
>> Ryan Lubke wrote:
>>> Okay, I've committed another set of refactorings to the
>>> FeedableBodyGenerator.
>>>
>>> For your use case, you should extend
>>> FeedableBodyGenerator.SimpleFeeder.
>>>
>>> Let me know if you run into issues.
>>>
>>>
>>>
>>> Sébastien Lorber wrote:
>>>> Yes I think it would, so that I can feed the queue at once
>>>>
>>>> One thread will be locked during the feeding for nothing
>>>> but it's not a real problem in my usecase.
>>>>
>>>>
>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>
>>>> Would having a different listener that will be notified
>>>> once async transferring has been started work better
>>>> for you?
>>>>
>>>> Something like:
>>>>
>>>> onAsyncTransferInitiated() {
>>>> // invoke your feed method
>>>> }
>>>>
>>>> ?
>>>>
>>>>
>>>>
>>>> Sébastien Lorber wrote:
>>>>> Unfortunatly I won't be able to use the Feeder
>>>>> non-blocking stuff for now, because of how the
>>>>> multipart request in handled in AHC
>>>>>
>>>>>
>>>>> Here's my feeding method:
>>>>>
>>>>> public void feed() throws IOException {
>>>>> Part[] partsArray = parts.toArray(new
>>>>> Part[parts.size()]);
>>>>> try ( OutputStream outputStream =
>>>>> createFeedingOutputStream() ) {
>>>>>
>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>> } catch (Exception e) {
>>>>> throw new IllegalStateException("Unable to feed
>>>>> the FeedableBodyGenerator",e);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> As you can see, the multipart Parts array can only be
>>>>> pushed to the OutputStream, I don't have any way to
>>>>> "pull" the data when the canFeed() method is triggered.
>>>>>
>>>>>
>>>>> But I've seen that there's
>>>>> a com.ning.http.multipart.MultipartBody#read that
>>>>> seems to provide a memory efficient way to pull data
>>>>> from a Multipart body...
>>>>>
>>>>> Should see what I come up with this
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> It seems the Feeder is highly recommended but not
>>>>> mandatory so I tried without.
>>>>>
>>>>> With my existing code it seems there is a
>>>>> synchronization problem.
>>>>>
>>>>>
>>>>> The feeding threads get locked to
>>>>> the prematureFeed.get();
>>>>>
>>>>> So the Grizzly kernel threads are unable to
>>>>> acquire the lock required to enter
>>>>> the initializeAsynchronousTransfer method
>>>>>
>>>>>
>>>>>
>>>>> Will try with an implementation of Feeder
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> Hmmm it seems I have a problem with one of
>>>>> your maven plugins. I'll try to bypass it, but
>>>>> for info:
>>>>>
>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>> [WARNING]
>>>>> [WARNING] Some problems were encountered while
>>>>> building the effective settings
>>>>> [WARNING]
>>>>> 'profiles.profile[default].repositories.repository.id
>>>>> <http://repositories.repository.id>' must be
>>>>> unique but found duplicate repository with id
>>>>> fullsix-maven-repository @
>>>>> /home/slorber/.m2/settings.xml
>>>>> [WARNING]
>>>>> [INFO] Scanning for projects...
>>>>> [INFO]
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Building Asynchronous Http Client
>>>>> 1.7.20-SNAPSHOT
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO]
>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean
>>>>> (default-clean) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>>> (enforce-maven) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>>> (enforce-versions) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-resources-plugin:2.4.3:resources
>>>>> (default-resources) @ async-http-client ---
>>>>> [INFO] Using 'UTF-8' encoding to copy filtered
>>>>> resources.
>>>>> [INFO] skip non existing resourceDirectory
>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>> [INFO]
>>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile
>>>>> (default-compile) @ async-http-client ---
>>>>> [INFO] Compiling 158 source files to
>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>> [INFO]
>>>>> *[INFO] ---
>>>>> animal-sniffer-maven-plugin:1.6:check
>>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>>> *[INFO] Checking unresolved references to
>>>>> org.codehaus.mojo.signature:java15:1.0*
>>>>> *[ERROR] Undefined reference:
>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V
>>>>> in
>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class*
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 8.747s
>>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>>> [INFO] Final Memory: 30M/453M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> *[ERROR] Failed to execute goal
>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>> (check-java-1.5-compat) on project
>>>>> async-http-client: Signature errors found.
>>>>> Verify them and put @IgnoreJRERequirement on
>>>>> them. -> [Help 1]*
>>>>> [ERROR]
>>>>> [ERROR] To see the full stack trace of the
>>>>> errors, re-run Maven with the -e switch.
>>>>> [ERROR] Re-run Maven using the -X switch to
>>>>> enable full debug logging.
>>>>> [ERROR]
>>>>> [ERROR] For more information about the errors
>>>>> and possible solutions, please read the
>>>>> following articles:
>>>>> [ERROR] [Help 1]
>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> Ok thank you, I'll try to implement that
>>>>> today and will give you my feedback :)
>>>>>
>>>>>
>>>>> 2013/9/10 Ryan Lubke
>>>>> <ryan.lubke_at_oracle.com
>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>
>>>>> Okay,
>>>>>
>>>>> I've committed my initial changes to
>>>>> the AHC repository. Here's a summary
>>>>> of the changes:
>>>>>
>>>>>
>>>>> *Improvements to the
>>>>> FeedableBodyGenerator (Grizzly's).
>>>>> - Don't allow queueing of data before
>>>>> initiateAsyncTransfer has been
>>>>> invoked. In low memory
>>>>> heaps, this could lead to an OOM if
>>>>> the source is feeding too fast. The
>>>>> new behavior is to
>>>>> block until initiateAsyncTransfer is
>>>>> called, at which time the blocked
>>>>> thread may proceed with
>>>>> the feed operation.
>>>>> - Introduce the concept of a Feeder.
>>>>> Implementations are responsible, at a
>>>>> high level, for:
>>>>> + letting the provider know that data
>>>>> is available to be fed without blocking
>>>>> + allowing the registration of a
>>>>> callback that the Feeder
>>>>> implementation may invoke
>>>>> to signal that more data is available,
>>>>> if it wasn't available at a previous
>>>>> point in time.
>>>>> - When using a Feeder with a secure
>>>>> request, the SSL handshake will be
>>>>> kicked off by the
>>>>> initiateAsyncTransfer call, but
>>>>> feeding of data will not occur until
>>>>> the handshake is complete.
>>>>> This is necessary as the SSLFilter
>>>>> will queue up all writes until the
>>>>> handshake is complete,
>>>>> and currently, the buffer isn't tied
>>>>> in with the transport flow control
>>>>> mechanism.
>>>>> NOTE: This new SSL behavior is not
>>>>> currently applied when invoking the
>>>>> feed() method
>>>>> outside the context of a Feeder. Still
>>>>> need to address that.
>>>>> - Exposed configuration of the async
>>>>> write queue limit through the
>>>>> FeedableBodyGenerator.
>>>>> This is an improvement on using a
>>>>> TransportCustomizer as any
>>>>> configuration there is
>>>>> transport-wide, and therefor applied
>>>>> to all Connections. By exposing it
>>>>> here, each feeder
>>>>> may have a different byte limit.
>>>>> - Improved documentation for this class*
>>>>>
>>>>>
>>>>>
>>>>> I recommend reading through the
>>>>> javadoc comments in the source [1] for
>>>>> FeedableBodyGenerator (comments welcome).
>>>>> Additionally, I would re-work your
>>>>> code to leverage the Feeder instead of
>>>>> calling feed() directly.
>>>>>
>>>>> If you have issues implementing
>>>>> Feeder, do let us know.
>>>>>
>>>>> If you have additional questions,
>>>>> again, let us know.
>>>>>
>>>>> Thanks,
>>>>> -rl
>>>>>
>>>>> [1]
>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>
>>>>>
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber wrote:
>>>>>>> So in the end I've end up with an
>>>>>>> implementation that's working for me.
>>>>>>>
>>>>>>>
>>>>>>> I think there are 2 bugs:
>>>>>>>
>>>>>>> 1) The bytes can accumulate in the
>>>>>>> FeedableBodyGenerator queue if the
>>>>>>> initialize(ctx) method is not called
>>>>>>> fast enough.
>>>>>>> This can be solved by using a
>>>>>>> BlockingQueue of size 1 and the
>>>>>>> put() method.
>>>>>>>
>>>>>>> 2) Once the context is injected, the
>>>>>>> FeedableBodyGenerator flushes the
>>>>>>> queue.
>>>>>>> The matter is that if the connection
>>>>>>> is new, not warmed up by a previous
>>>>>>> request, then the SSL handshake is
>>>>>>> not done yet, and it seems that the
>>>>>>> bytes are accumulated in some part
>>>>>>> of the SSL filter which doesn't
>>>>>>> deliver them to the connection until
>>>>>>> the handshake has completed,
>>>>>>> so c.canWrite() continues to return
>>>>>>> true.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have replaced some part of the
>>>>>>> FeedableBodyGenerator to test this
>>>>>>> and it works pretty fine. See what I
>>>>>>> have changed:
>>>>>>>
>>>>>>> 1)
>>>>>>> private final
>>>>>>> BlockingQueue<BodyPart> queue = new
>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>
>>>>>>>
>>>>>>> 2)
>>>>>>> public void feed(final Buffer
>>>>>>> buffer, final boolean last) throws
>>>>>>> IOException {
>>>>>>> try {
>>>>>>> queue.put(new
>>>>>>> BodyPart(buffer, last));
>>>>>>> } catch
>>>>>>> (InterruptedException e) {
>>>>>>> throw new RuntimeException(e);
>>>>>>> }
>>>>>>> queueSize.incrementAndGet();
>>>>>>> if (context != null) {
>>>>>>>
>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private void
>>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>>>>> fcc) {
>>>>>>> if (
>>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>>> while (
>>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>>> try { Thread.sleep(10); }
>>>>>>> catch ( Exception e ) { throw new
>>>>>>> RuntimeException(e); }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private boolean
>>>>>>> connectionIsReadyToWrite(FilterChainContext
>>>>>>> fcc) {
>>>>>>> Connection connection =
>>>>>>> fcc.getConnection();
>>>>>>> SSLEngine sslEngine =
>>>>>>> SSLUtils.getSSLEngine(connection);
>>>>>>> return sslEngine != null &&
>>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think?
>>>>>>
>>>>>> We had come to similar conclusions on
>>>>>> this end. I'm still working through
>>>>>> testing the idea I mentioned
>>>>>> previously (took longer than I
>>>>>> expected - sorry).
>>>>>> I hope to have something for you to
>>>>>> test very soon.
>>>>>>
>>>>>> Note that it will be taking the above
>>>>>> into account as well.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>>
>>>>>>> I have tried to put a while (
>>>>>>> context == null ) Thread.sleep
>>>>>>> but it doesn't seem to work,
>>>>>>> when the context gets injected,
>>>>>>> after the sleeps, there's an OOM
>>>>>>>
>>>>>>> So I hope you'll have more
>>>>>>> success with your alternative :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have done another test,
>>>>>>> remember my code that worked,
>>>>>>> which previously "warmed" the
>>>>>>> Thread with an useless request.
>>>>>>>
>>>>>>> private Runnable
>>>>>>> uploadSingleDocumentRunnable =
>>>>>>> new Runnable() {
>>>>>>> @Override
>>>>>>> public void run() {
>>>>>>> try {
>>>>>>> getUselessSessionCode();
>>>>>>> Thread.sleep(X);
>>>>>>> uploadSingleDocument();
>>>>>>> } catch ( Exception e ) {
>>>>>>> throw new
>>>>>>> RuntimeException("file upload
>>>>>>> failed",e);
>>>>>>> }
>>>>>>> }
>>>>>>> };
>>>>>>>
>>>>>>> I have put a sleep of X between
>>>>>>> the useless warmup request, and
>>>>>>> the real upload request
>>>>>>>
>>>>>>>
>>>>>>> What I noticed is that there is
>>>>>>> a very different behavior
>>>>>>> according to the value of X
>>>>>>>
>>>>>>>
>>>>>>> Under 10 seconds, it seems the
>>>>>>> stuff is still warm, I can
>>>>>>> upload the documents.
>>>>>>> Around 10 seconds I get a stack
>>>>>>> which seems to be "connection
>>>>>>> closed" or something
>>>>>>> Above 10 seconds, I get OOM like
>>>>>>> if the stuff wasn't warm.
>>>>>>>
>>>>>>>
>>>>>>> The stacks I get for 10 seconds
>>>>>>> looks like
>>>>>>>
>>>>>>> Caused by:
>>>>>>> javax.net.ssl.SSLException:
>>>>>>> SSLEngine is CLOSED
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>
>>>>>>> I think I got some other
>>>>>>> different stacks saying
>>>>>>> Connection Closed Remotely or
>>>>>>> something like that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So it seems that something is
>>>>>>> bound to my thread, and it stays
>>>>>>> bound to it for about 10
>>>>>>> seconds, do you have any idea
>>>>>>> what it could be?
>>>>>>> (My connection timeout setting
>>>>>>> seems to have no effect on this
>>>>>>> 10s threshold)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Ryan Lubke
>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>
>>>>>>> That is one solution. I'm
>>>>>>> working out an alternative
>>>>>>> right now. Stay tuned!
>>>>>>>
>>>>>>>>
>>>>>>>> Anyway it's not a problem,
>>>>>>>> I think the
>>>>>>>> FeedableBodyGenerator.feed() method
>>>>>>>> just has to block until a
>>>>>>>> context has been (and
>>>>>>>> ThreadCache initialized) to
>>>>>>>> avoid OOM errors
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What is very strange is
>>>>>>>> that I tested
>>>>>>>> with/without the same
>>>>>>>> sessionCode with our
>>>>>>>> previous code, the one
>>>>>>>> not using
>>>>>>>> FeedableBodyGenerator,
>>>>>>>> which has a high memory
>>>>>>>> consumption.
>>>>>>>> Despites the fact it
>>>>>>>> had high memory
>>>>>>>> consumption, it seems
>>>>>>>> work fine to upload
>>>>>>>> multiple documents if
>>>>>>>> allocated with a large
>>>>>>>> heap, and the
>>>>>>>> sessionCode seems to
>>>>>>>> have no effect.
>>>>>>>>
>>>>>>>> On the new impl using
>>>>>>>> the
>>>>>>>> FeedableBodyGenerator,
>>>>>>>> the sessionCode sent as
>>>>>>>> a multipart bodypart
>>>>>>>> seems to have an effect.
>>>>>>>>
>>>>>>>> I have tried to feed
>>>>>>>> the queue before
>>>>>>>> sending the request to
>>>>>>>> AHC, but this leads to
>>>>>>>> this exception
>>>>>>>> (with/without
>>>>>>>> sessionCode switching)
>>>>>>>> Caused by:
>>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>>> Timeout exceeded
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien
>>>>>>>> Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>> By the way, by
>>>>>>>> using a low timeout
>>>>>>>> with the same
>>>>>>>> sessioncode, I got
>>>>>>>> the following NPE:
>>>>>>>>
>>>>>>>> Caused by:
>>>>>>>> java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>
>>>>>>>>
>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>> httpCtx =
>>>>>>>>
>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>
>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>
>>>>>>>> I guess the httpCtx
>>>>>>>> is not already
>>>>>>>> available to be aborted
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien
>>>>>>>> Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is right,
>>>>>>>> here's a log I
>>>>>>>> have when I use
>>>>>>>> the same
>>>>>>>> session code,
>>>>>>>> ie the remote
>>>>>>>> host is
>>>>>>>> blocking the
>>>>>>>> data or something.
>>>>>>>> This is
>>>>>>>> obtained by
>>>>>>>> running 5
>>>>>>>> parallel uploads.
>>>>>>>>
>>>>>>>> *Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> *Flushing queue
>>>>>>>> of size 97 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> *Flushing queue
>>>>>>>> of size 100
>>>>>>>> with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> *Flushing queue
>>>>>>>> of size 160
>>>>>>>> with
>>>>>>>> allowBlocking =
>>>>>>>> true*
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>> GC overhead
>>>>>>>> limit exceeded
>>>>>>>> Dumping heap to
>>>>>>>> /ome/lorber/ureau/om
>>>>>>>> ...
>>>>>>>> Unable to
>>>>>>>> create
>>>>>>>> /ome/lorber/ureau/om:
>>>>>>>> Le fichier existe
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Disconnected
>>>>>>>> from the target
>>>>>>>> VM, address:
>>>>>>>> '127.0.0.1:49268 <http://127.0.0.1:49268>',
>>>>>>>> transport: 'socket'
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Otherwise, with
>>>>>>>> different
>>>>>>>> session codes,
>>>>>>>> I get the
>>>>>>>> following:
>>>>>>>>
>>>>>>>> *Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> *Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> *Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> *Flushing queue
>>>>>>>> of size 0 with
>>>>>>>> allowBlocking =
>>>>>>>> false*
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> Flushing queue
>>>>>>>> of size 1 with
>>>>>>>> allowBlocking =
>>>>>>>> true
>>>>>>>> ... and this
>>>>>>>> continues
>>>>>>>> without OOM
>>>>>>>>
>>>>>>>>
>>>>>>>> So, this seems
>>>>>>>> to be the problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I think it
>>>>>>>> would be great
>>>>>>>> to be able to
>>>>>>>> be able to
>>>>>>>> choose the
>>>>>>>> queue impl
>>>>>>>> behind
>>>>>>>> that FeedableBodyGenerator,
>>>>>>>> like I
>>>>>>>> suggested in my
>>>>>>>> pull request.
>>>>>>>>
>>>>>>>> See here:
>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>
>>>>>>>> Using a
>>>>>>>> LinkedBlockingQueue
>>>>>>>> seems to be a
>>>>>>>> nice idea in
>>>>>>>> this context,
>>>>>>>> and in my case
>>>>>>>> I would
>>>>>>>> probably use a
>>>>>>>> queue of size 1
>>>>>>>>
>>>>>>>>
>>>>>>>> This would
>>>>>>>> handle the
>>>>>>>> blocking of the
>>>>>>>> feed method,
>>>>>>>> without having
>>>>>>>> to use this:
>>>>>>>> if
>>>>>>>> (context != null) {
>>>>>>>>
>>>>>>>> flushQueue(true);
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Or perhaps the
>>>>>>>> feed() method
>>>>>>>> have to wait
>>>>>>>> until a context
>>>>>>>> is set in the
>>>>>>>> BodyGenerator ?
>>>>>>>>
>>>>>>>>
>>>>>>>> I think it
>>>>>>>> would be more
>>>>>>>> clear if
>>>>>>>> the initializeAsynchronousTransfer
>>>>>>>> simply didn't
>>>>>>>> flush the queue
>>>>>>>> but just setup
>>>>>>>> the context.
>>>>>>>> Then the feed
>>>>>>>> method would
>>>>>>>> block until
>>>>>>>> there's a
>>>>>>>> context set,
>>>>>>>> and then flush
>>>>>>>> the queue with
>>>>>>>> blocking behavior.
>>>>>>>>
>>>>>>>> This is
>>>>>>>> probably the
>>>>>>>> next step, but
>>>>>>>> as we are using
>>>>>>>> AHC for async,
>>>>>>>> it would
>>>>>>>> probably be
>>>>>>>> great if that
>>>>>>>> blocking feed()
>>>>>>>> method was
>>>>>>>> called in a
>>>>>>>> worker thread
>>>>>>>> instead of our
>>>>>>>> main thread.
>>>>>>>> I won't use
>>>>>>>> this but
>>>>>>>> someone who
>>>>>>>> really wants a
>>>>>>>> non-blocking
>>>>>>>> impl of
>>>>>>>> performant
>>>>>>>> multipart
>>>>>>>> fileupload
>>>>>>>> would probably
>>>>>>>> need this, or
>>>>>>>> will use an
>>>>>>>> ExecutorService
>>>>>>>> for the feeding
>>>>>>>> operations as a
>>>>>>>> workaround.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks again
>>>>>>>> for your reactivity
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/4 Ryan
>>>>>>>> Lubke
>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien
>>>>>>>> Lorber wrote:
>>>>>>>>> I've
>>>>>>>>> integrated
>>>>>>>>> this
>>>>>>>>> change and
>>>>>>>>> it works
>>>>>>>>> fine
>>>>>>>>> except a
>>>>>>>>> little detail.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm
>>>>>>>>> uploading
>>>>>>>>> files to a
>>>>>>>>> third
>>>>>>>>> party API
>>>>>>>>> (a bit
>>>>>>>>> like S3).
>>>>>>>>> This API
>>>>>>>>> requires a
>>>>>>>>> "sessionCode"
>>>>>>>>> in each
>>>>>>>>> request.
>>>>>>>>> So there
>>>>>>>>> is a
>>>>>>>>> multipart
>>>>>>>>> StringPart
>>>>>>>>> with that
>>>>>>>>> SessionCode.
>>>>>>>>>
>>>>>>>>> We used to
>>>>>>>>> have a
>>>>>>>>> cache
>>>>>>>>> which
>>>>>>>>> holds the
>>>>>>>>> sessionCode 30min
>>>>>>>>> per user
>>>>>>>>> so that we
>>>>>>>>> do not
>>>>>>>>> need to
>>>>>>>>> init a new
>>>>>>>>> session
>>>>>>>>> each time.
>>>>>>>>>
>>>>>>>>> I had
>>>>>>>>> troubles
>>>>>>>>> in this
>>>>>>>>> very
>>>>>>>>> specific
>>>>>>>>> case: when
>>>>>>>>> I upload 5
>>>>>>>>> docs with
>>>>>>>>> the same
>>>>>>>>> session code.
>>>>>>>>> When I
>>>>>>>>> remove the
>>>>>>>>> cache and
>>>>>>>>> use 5
>>>>>>>>> different
>>>>>>>>> session
>>>>>>>>> codes, it
>>>>>>>>> works fine.
>>>>>>>>>
>>>>>>>>> I guess
>>>>>>>>> the remote
>>>>>>>>> service is
>>>>>>>>> blocking
>>>>>>>>> concurrent
>>>>>>>>> uploads
>>>>>>>>> with the
>>>>>>>>> same
>>>>>>>>> session
>>>>>>>>> code. I
>>>>>>>>> don't know
>>>>>>>>> at all.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Where I
>>>>>>>>> want to go
>>>>>>>>> is that I
>>>>>>>>> wouldn't
>>>>>>>>> have
>>>>>>>>> expected
>>>>>>>>> Grizzly to OOM
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Avertissement:
>>>>>>>>> Exception
>>>>>>>>> during
>>>>>>>>> FilterChain execution
>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>> Java heap
>>>>>>>>> space
>>>>>>>>> at
>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>> at
>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Caused by:
>>>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>>>> null
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> multipart.body.generator.feeder.buffer=100000
>>>>>>>>> -> size of
>>>>>>>>> each
>>>>>>>>> Buffer
>>>>>>>>> sent to
>>>>>>>>> the
>>>>>>>>> FeedableBodyGenerator
>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>> (I tried
>>>>>>>>> other
>>>>>>>>> settings,
>>>>>>>>> including
>>>>>>>>> AUTO_SIZE)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you
>>>>>>>>> have any
>>>>>>>>> idea why
>>>>>>>>> is there
>>>>>>>>> an OOM
>>>>>>>>> with these
>>>>>>>>> settings?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Perhaps it
>>>>>>>>> is because
>>>>>>>>> the feed()
>>>>>>>>> method of
>>>>>>>>> FeedableBodyGenerator
>>>>>>>>> doesn't
>>>>>>>>> block
>>>>>>>>> until the
>>>>>>>>> context is
>>>>>>>>> initialized.
>>>>>>>>> I guess
>>>>>>>>> the initializeAsynchronousTransfer
>>>>>>>>> is called
>>>>>>>>> only once
>>>>>>>>> the
>>>>>>>>> connection
>>>>>>>>> is
>>>>>>>>> established,
>>>>>>>>> and
>>>>>>>>> perhaps a
>>>>>>>>> lot of
>>>>>>>>> Buffer are
>>>>>>>>> added to
>>>>>>>>> the queue...
>>>>>>>> Yes, it's
>>>>>>>> invoked
>>>>>>>> once the
>>>>>>>> request has
>>>>>>>> been
>>>>>>>> dispatched,
>>>>>>>> so if the
>>>>>>>> generator
>>>>>>>> is fed a
>>>>>>>> lot before
>>>>>>>> the
>>>>>>>> request, I
>>>>>>>> could see
>>>>>>>> this happening.
>>>>>>>> I'll see
>>>>>>>> what I can
>>>>>>>> do to
>>>>>>>> alleviate
>>>>>>>> that case.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> But I'm
>>>>>>>>> not sure
>>>>>>>>> at all
>>>>>>>>> because
>>>>>>>>> the
>>>>>>>>> session
>>>>>>>>> code is
>>>>>>>>> transmitted as
>>>>>>>>> a BodyPart
>>>>>>>>> and I get
>>>>>>>>> the same
>>>>>>>>> problem if
>>>>>>>>> i put it
>>>>>>>>> as the
>>>>>>>>> first or
>>>>>>>>> last
>>>>>>>>> multipart.
>>>>>>>>>
>>>>>>>>> It's not a
>>>>>>>>> big deal,
>>>>>>>>> perhaps I
>>>>>>>>> should
>>>>>>>>> always use
>>>>>>>>> a
>>>>>>>>> different
>>>>>>>>> session
>>>>>>>>> code for
>>>>>>>>> concurrent
>>>>>>>>> operations
>>>>>>>>> but I'd
>>>>>>>>> like to be
>>>>>>>>> sure that
>>>>>>>>> we won't
>>>>>>>>> have this
>>>>>>>>> issue in
>>>>>>>>> production...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/3
>>>>>>>>> Ryan Lubke
>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>
>>>>>>>>> Good
>>>>>>>>> catch. Fixed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sébastien
>>>>>>>>> Lorber
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's
>>>>>>>>>> a
>>>>>>>>>> little mistake
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> grizzly
>>>>>>>>>> ahc
>>>>>>>>>> provider
>>>>>>>>>> relative
>>>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>> write
>>>>>>>>>> queue
>>>>>>>>>> size.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>
>>>>>>>>>> As
>>>>>>>>>> you
>>>>>>>>>> can
>>>>>>>>>> see,
>>>>>>>>>> the
>>>>>>>>>> TransportCustomizer
>>>>>>>>>> is
>>>>>>>>>> called,
>>>>>>>>>> and
>>>>>>>>>> then
>>>>>>>>>> the
>>>>>>>>>> write
>>>>>>>>>> queue
>>>>>>>>>> size
>>>>>>>>>> (among other
>>>>>>>>>> things)
>>>>>>>>>> is
>>>>>>>>>> set
>>>>>>>>>> to
>>>>>>>>>> AUTO_SIZE
>>>>>>>>>> (instead
>>>>>>>>>> of
>>>>>>>>>> previously
>>>>>>>>>> UNLIMITED)
>>>>>>>>>>
>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>> think
>>>>>>>>>> the
>>>>>>>>>> default
>>>>>>>>>> settings
>>>>>>>>>> like
>>>>>>>>>> this
>>>>>>>>>> AUTO_SIZE
>>>>>>>>>> attribute
>>>>>>>>>> should be
>>>>>>>>>> set
>>>>>>>>>> before the
>>>>>>>>>> customization
>>>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>> transport,
>>>>>>>>>> or
>>>>>>>>>> they
>>>>>>>>>> would
>>>>>>>>>> override
>>>>>>>>>> the
>>>>>>>>>> value
>>>>>>>>>> we
>>>>>>>>>> customized.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This
>>>>>>>>>> is
>>>>>>>>>> actually
>>>>>>>>>> my
>>>>>>>>>> case,
>>>>>>>>>> since
>>>>>>>>>> I
>>>>>>>>>> can't
>>>>>>>>>> reproduce
>>>>>>>>>> my
>>>>>>>>>> "bug"
>>>>>>>>>> which
>>>>>>>>>> is
>>>>>>>>>> "high
>>>>>>>>>> memory consumption",
>>>>>>>>>> even
>>>>>>>>>> when
>>>>>>>>>> using
>>>>>>>>>> -1 /
>>>>>>>>>> UNLIMITED
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This
>>>>>>>>>> could
>>>>>>>>>> work
>>>>>>>>>> fine
>>>>>>>>>> for
>>>>>>>>>> me
>>>>>>>>>> with
>>>>>>>>>> AUTO_SIZE,
>>>>>>>>>> but
>>>>>>>>>> I'd
>>>>>>>>>> rather be
>>>>>>>>>> able
>>>>>>>>>> to
>>>>>>>>>> tune
>>>>>>>>>> this
>>>>>>>>>> parameter
>>>>>>>>>> during load
>>>>>>>>>> tests
>>>>>>>>>> to
>>>>>>>>>> see
>>>>>>>>>> the
>>>>>>>>>> effect.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/31
>>>>>>>>>> Sebastien
>>>>>>>>>> Lorber <lorber.sebastien_at_gmail.com
>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> i
>>>>>>>>>> will
>>>>>>>>>> ckeck
>>>>>>>>>> that
>>>>>>>>>> on monday.
>>>>>>>>>> I
>>>>>>>>>> can
>>>>>>>>>> now
>>>>>>>>>> upload
>>>>>>>>>> a
>>>>>>>>>> 500m
>>>>>>>>>> file
>>>>>>>>>> with
>>>>>>>>>> 40m
>>>>>>>>>> heap
>>>>>>>>>> size
>>>>>>>>>> ;)
>>>>>>>>>>
>>>>>>>>>> Envoyé
>>>>>>>>>> de mon
>>>>>>>>>> iPhone
>>>>>>>>>>
>>>>>>>>>> Le 30
>>>>>>>>>> août
>>>>>>>>>> 2013
>>>>>>>>>> à
>>>>>>>>>> 20:49,
>>>>>>>>>> Ryan
>>>>>>>>>> Lubke
>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>>> I'm
>>>>>>>>>>> going
>>>>>>>>>>> to
>>>>>>>>>>> be
>>>>>>>>>>> updating
>>>>>>>>>>> the
>>>>>>>>>>> Grizzly
>>>>>>>>>>> provider
>>>>>>>>>>> such
>>>>>>>>>>> that
>>>>>>>>>>> AUTO_SIZE
>>>>>>>>>>> (not
>>>>>>>>>>> AUTO_TUNE)
>>>>>>>>>>> is
>>>>>>>>>>> the
>>>>>>>>>>> default,
>>>>>>>>>>> so
>>>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>> avoid
>>>>>>>>>>> the
>>>>>>>>>>> use
>>>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Regarding
>>>>>>>>>>>> your
>>>>>>>>>>>> tuning
>>>>>>>>>>>> question,
>>>>>>>>>>>> I
>>>>>>>>>>>> would
>>>>>>>>>>>> probably
>>>>>>>>>>>> set
>>>>>>>>>>>> the
>>>>>>>>>>>> value
>>>>>>>>>>>> to
>>>>>>>>>>>> AsyncQueueWriter.AUTO_TUNE
>>>>>>>>>>>> (this
>>>>>>>>>>>> will
>>>>>>>>>>>> be
>>>>>>>>>>>> four
>>>>>>>>>>>> times
>>>>>>>>>>>> the
>>>>>>>>>>>> socket
>>>>>>>>>>>> write
>>>>>>>>>>>> buffer)
>>>>>>>>>>>> and
>>>>>>>>>>>> see
>>>>>>>>>>>> how
>>>>>>>>>>>> that
>>>>>>>>>>>> works.
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>> Lubke
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> A question
>>>>>>>>>>>>> first.
>>>>>>>>>>>>> With
>>>>>>>>>>>>> these
>>>>>>>>>>>>> changes,
>>>>>>>>>>>>> your
>>>>>>>>>>>>> memory
>>>>>>>>>>>>> usage
>>>>>>>>>>>>> is
>>>>>>>>>>>>> more
>>>>>>>>>>>>> inline
>>>>>>>>>>>>> with
>>>>>>>>>>>>> what
>>>>>>>>>>>>> you
>>>>>>>>>>>>> were
>>>>>>>>>>>>> looking
>>>>>>>>>>>>> for?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> By
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> idea
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> 1.7.20
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> released
>>>>>>>>>>>>>> (with
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>> improvements?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> know
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> a release
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> install
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>> own
>>>>>>>>>>>>>> temp
>>>>>>>>>>>>>> release
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> Nexus
>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/30
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank
>>>>>>>>>>>>>> you,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> works
>>>>>>>>>>>>>> fine!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just
>>>>>>>>>>>>>> had
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> modify
>>>>>>>>>>>>>> a single
>>>>>>>>>>>>>> line
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> commit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>> I let
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> (-1)
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>> block,
>>>>>>>>>>>>>> canWrite
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> returns
>>>>>>>>>>>>>> true
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Btw,
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>> find
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> pass
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> a config
>>>>>>>>>>>>>> attribute,
>>>>>>>>>>>>>> neither
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> size
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> write
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> talked
>>>>>>>>>>>>>> about.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> end,
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> current
>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> "canWrite
>>>>>>>>>>>>>> = false"
>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>> not,
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> please
>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> v1.7.20
>>>>>>>>>>>>>> ? thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>> does
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>> sens
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> number
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>> un
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> feed(Buffer)
>>>>>>>>>>>>>> method
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>> ? do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> tuning
>>>>>>>>>>>>>> recommandation?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please
>>>>>>>>>>>>>> disregard.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>> a sample
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>> performing
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> feed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>>> recommend
>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> Connection.canWrite()
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener)
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>>> default,
>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> four
>>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> buffer).
>>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> exceeds
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> value,
>>>>>>>>>>>>>>>> canWrite()
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>> false.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> occurs,
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>>> a WriteListener
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> notified
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> configured
>>>>>>>>>>>>>>>> max
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> simulate
>>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>>> callback
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>> invoked.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> FutureImpl<Boolean>
>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>> = Futures.createSafeFuture();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> //
>>>>>>>>>>>>>>>> Connection
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> obtained
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> calling
>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> connection.notifyCanWrite(new
>>>>>>>>>>>>>>>> WriteHandler()
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>>> throws
>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>>> onError(Throwable
>>>>>>>>>>>>>>>> t)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>>> writeTimeout
>>>>>>>>>>>>>>>> = 30;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>>> (ExecutionException
>>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>>> (Exception
>>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> did
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> a blocking
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> totally
>>>>>>>>>>>>>>>>> useless
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>> consuming
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>> blocks
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> feeding,
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> intention
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> beginning.
>>>>>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>>> strategy
>>>>>>>>>>>>>>>>> used?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I use
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> use SameThreadIOStrategy
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>> a 70m
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 50m,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> put
>>>>>>>>>>>>>>>>> a Thread.sleep(30)
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> I know.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>>> things
>>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send.
>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> possible?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> 100mo
>>>>>>>>>>>>>>>>> heap,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>> a sleep
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 100ms
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>>>> chance
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> MessageCloner
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>> filter?
>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> cloned,
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> I remove
>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>> How?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> trying
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> a 500m
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> cases
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> consumption
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> x parallel
>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> according
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> activity.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>> reduced
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> footprint
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>> allocation
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> concerned,
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> untunable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That
>>>>>>>>>>>>>>>>> said,
>>>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>>> free
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>> a feature
>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>>> size.
>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>> application
>>>>>>>>>>>>>>>>> specific.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> curious,
>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> sending?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> buffers
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a size
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> 33842
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> limit
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> near
>>>>>>>>>>>>>>>>>> half
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> capacity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> tune
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> consumes
>>>>>>>>>>>>>>>>>> less
>>>>>>>>>>>>>>>>>> memory?
>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> ideal
>>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> reviewing
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>>> today,
>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>> again!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> OOM:
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> stands
>>>>>>>>>>>>>>>>>> now,
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> SSLFilter,
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>> a buffer
>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> order
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> accommodate
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> encrypted
>>>>>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Depending
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>> endpoints,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> becoming
>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> situation,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>> pretty
>>>>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>>>>> shielded
>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>> I will
>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> leverage
>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> With
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>> usecase
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> works,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> uploaded
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>>> a big
>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> body
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> that?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>>> memory,
>>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>>>> of:
>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> SSLUtils$3
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> BufferBuffers
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> WriteResult
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> exemple
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>>> stacktrace:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>>>>>> idea?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Excellent!
>>>>>>>>>>>>>>>>>>> Looking
>>>>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>>> request!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> works
>>>>>>>>>>>>>>>>>>>> fine,
>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>> a pull
>>>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> tomorrow
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> a better
>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>> Part
>>>>>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I created
>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> OutputStream
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> redirects
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>>>>> feeder.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>> I currently
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> feeder
>>>>>>>>>>>>>>>>>>>> feeds
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>>>> polling
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>> I need
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> expose
>>>>>>>>>>>>>>>>>>>> a limit
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> something,
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> a thread
>>>>>>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> filepart
>>>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes,
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>> I was
>>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> tackle
>>>>>>>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>>>>>> up
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok
>>>>>>>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think
>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>>> do,
>>>>>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>>> bodyGenerator
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Request
>>>>>>>>>>>>>>>>>>>>> uploadRequest1
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response>
>>>>>>>>>>>>>>>>>>>>> asyncRes
>>>>>>>>>>>>>>>>>>>>> = asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Response
>>>>>>>>>>>>>>>>>>>>> uploadResponse
>>>>>>>>>>>>>>>>>>>>> = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I guess
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> interesting
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/26
>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> / Grizzly
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> streaming,
>>>>>>>>>>>>>>>>>>>>> I mean
>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>>>>> a byte[]
>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> contans
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file,
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>>> a large
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>>>>> tryied
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> a Heap
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> ByteBuffer
>>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>> managers,
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> reallocate=true/false
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> success.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> content
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> appended
>>>>>>>>>>>>>>>>>>>>> wto
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> BufferOutputStream,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> underlying
>>>>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>>> uploading
>>>>>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> Grizzly?
>>>>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>>>>>>>> tied
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> Multipart
>>>>>>>>>>>>>>>>>>>>> uploads,
>>>>>>>>>>>>>>>>>>>>> you'd
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> convert
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> multipart
>>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> Buffers
>>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>> a closer
>>>>>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> area
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Btw
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> upload.
>>>>>>>>>>>>>>>>>>>>> I receive
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> transmit
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> a storage
>>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>>> (like
>>>>>>>>>>>>>>>>>>>>> S3).
>>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>> bevause
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> streaming
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>>>>>> fle
>>>>>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> system,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> provides
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> input
>>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>