Re: Upload a large file without oom with Grizzly

From: Ryan Lubke <ryan.lubke_at_oracle.com>
Date: Thu, 12 Sep 2013 10:08:01 -0700

Sébastien Lorber wrote:
>
>
>
> I noticed something strange.
> On the FileInputStream I have, I've added a log on the close() of the
> stream which is called once the whole file has been read to be sent to
> the feeder.
>
> 1) If I perform a request (ie my session init request in the previous
> discussions) before doing my multipart upload, the thread that does
> execute the feeding is the thread that fires the request, and not a
> Grizzly worker.
> [Thread-30] INFO Will close file stream
> java.io.FileInputStream_at_27fe4315 after having read 1440304 bytes
>
>
> 2) If I don't do any request before firing the multipart upload, the
> thread that does the feeding is a Grizzly threadpool worker thread:
> [Grizzly(22)] INFO Will close file stream
> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>
>
> Is this normal? I would expect a worker thread to always be used, and
> the main applicative thread that performs the request to never be
> blocking. (but it's not such an issue for me, we don't have a reactive
> non-blocking app anyway)
It's currently expected behavior. Will need to re-evaluate this based
on the new semantics of the FeedableBodyGenerator.
>
>
> On the 1st case, here's the stacktrace when the feed method is called:
>
> *at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)*
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
> at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
> *at
> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)*
>
> So it seems this happen when the handshake has already been done when
> *initializeAsynchronousTransfer* is called, so that we do not go
> through the HandshakeListener
>
>
> Notice that this case seems to also happen in a few threads on the 2nd
> case, but most of the threads are Grizzly workers.
>
>
>
>
>
>
>
>
> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com
> <mailto:lorber.sebastien_at_gmail.com>>
>
>
>
> Hi.
>
>
> Thanks, it seems to work.
>
> I would suggest to throw IOException on the flush method since the
> feed method is supposed to be called here
>
>
> My implementation of flush is:
>
> @Override
> public void flush() {
> Part[] partsArray = parts.toArray(new Part[parts.size()]);
> try ( OutputStream outputStream =
> createFeedingOutputStream() ) {
> Part.sendParts(outputStream,partsArray,multipartBoundary);
> } catch (Exception e) {
> throw new IllegalStateException("Unable to feed the
> FeedableBodyGenerator",e);
> }
> }
>
> Is this correct? The OutputStream redirects the bytes written to
> the feed(Buffer) method
>
>
>
>
>
>
> There seems to be some concurrency issue. Because the upload of 1
> file seems fine, but when using multiple threads, I often get the
> following stack:
> Caused by: java.io.IOException: Stream Closed
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:242)
> at
> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
> at com.ning.http.multipart.Part.send(Part.java:331)
> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>
>
> This is because the flush() method is called multiple times for
> the same request on some cases.
> I guess this is not supposed to happen.
> What I understand is that the flush() method is supposed to be
> called only once.
>
>
> Using debug logging breakpoints I get the following:
>
> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 0 started
> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 1 started
> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 2 started
> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 3 started
> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 4 started
> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 5 started
> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 6 started
> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 7 started
> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 8 started
> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 9 started
> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 10
> started
> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 11
> started
> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 12
> started
> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 13
> started
> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 14
> started
> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 15
> started
> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 16
> started
> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 17
> started
> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 18
> started
> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread 19
> started
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
> Adding handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
> *Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
> *Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
> *Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
> *Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
> *Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
> Completing and removing handshake listener for
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>
>
> As you can see the same HandshakeListener seems to be called
> multiple times for the same FeedableBodyGenerator
>
> When this happens, the stack is:
>
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
> at
> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
> ~[grizzly-framework-2.3.5.jar:2.3.5]
>
>
>
>
> So I tried with the following code:
>
> private boolean alreadyFlushed = false;
> @Override
> public synchronized void flush() {
> if ( alreadyFlushed ) {
> return;
> }
> startFeeding();
> alreadyFlushed = true;
> }
>
> It works fine when upload small files.
>
> But with larger files, I often get TimeoutException stacks for
> some of the threads:
>
> Caused by: java.util.concurrent.TimeoutException
> at
> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
> at
> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>
>
>
>
> Did I miss something?
>
>
>
>
>
>
> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com
> <mailto:ryan.lubke_at_oracle.com>>
>
> Committed another small change. Please make sure you're at
> the latest when you build.
>
> -rl
>
>
> Ryan Lubke wrote:
>> Okay, I've committed another set of refactorings to the
>> FeedableBodyGenerator.
>>
>> For your use case, you should extend
>> FeedableBodyGenerator.SimpleFeeder.
>>
>> Let me know if you run into issues.
>>
>>
>>
>> Sébastien Lorber wrote:
>>> Yes I think it would, so that I can feed the queue at once
>>>
>>> One thread will be locked during the feeding for nothing but
>>> it's not a real problem in my usecase.
>>>
>>>
>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
>>> <mailto:ryan.lubke_at_oracle.com>>
>>>
>>> Would having a different listener that will be notified
>>> once async transferring has been started work better for
>>> you?
>>>
>>> Something like:
>>>
>>> onAsyncTransferInitiated() {
>>> // invoke your feed method
>>> }
>>>
>>> ?
>>>
>>>
>>>
>>> Sébastien Lorber wrote:
>>>> Unfortunatly I won't be able to use the Feeder
>>>> non-blocking stuff for now, because of how the
>>>> multipart request in handled in AHC
>>>>
>>>>
>>>> Here's my feeding method:
>>>>
>>>> public void feed() throws IOException {
>>>> Part[] partsArray = parts.toArray(new
>>>> Part[parts.size()]);
>>>> try ( OutputStream outputStream =
>>>> createFeedingOutputStream() ) {
>>>>
>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>> } catch (Exception e) {
>>>> throw new IllegalStateException("Unable to feed
>>>> the FeedableBodyGenerator",e);
>>>> }
>>>> }
>>>>
>>>>
>>>> As you can see, the multipart Parts array can only be
>>>> pushed to the OutputStream, I don't have any way to
>>>> "pull" the data when the canFeed() method is triggered.
>>>>
>>>>
>>>> But I've seen that there's
>>>> a com.ning.http.multipart.MultipartBody#read that seems
>>>> to provide a memory efficient way to pull data from a
>>>> Multipart body...
>>>>
>>>> Should see what I come up with this
>>>>
>>>>
>>>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>> It seems the Feeder is highly recommended but not
>>>> mandatory so I tried without.
>>>>
>>>> With my existing code it seems there is a
>>>> synchronization problem.
>>>>
>>>>
>>>> The feeding threads get locked to
>>>> the prematureFeed.get();
>>>>
>>>> So the Grizzly kernel threads are unable to acquire
>>>> the lock required to enter
>>>> the initializeAsynchronousTransfer method
>>>>
>>>>
>>>>
>>>> Will try with an implementation of Feeder
>>>>
>>>>
>>>>
>>>> 2013/9/10 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>> Hmmm it seems I have a problem with one of your
>>>> maven plugins. I'll try to bypass it, but for info:
>>>>
>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>> [WARNING]
>>>> [WARNING] Some problems were encountered while
>>>> building the effective settings
>>>> [WARNING]
>>>> 'profiles.profile[default].repositories.repository.id
>>>> <http://repositories.repository.id>' must be
>>>> unique but found duplicate repository with id
>>>> fullsix-maven-repository @
>>>> /home/slorber/.m2/settings.xml
>>>> [WARNING]
>>>> [INFO] Scanning for projects...
>>>> [INFO]
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Building Asynchronous Http Client
>>>> 1.7.20-SNAPSHOT
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO]
>>>> [INFO] --- maven-clean-plugin:2.4.1:clean
>>>> (default-clean) @ async-http-client ---
>>>> [INFO]
>>>> [INFO] ---
>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>> (enforce-maven) @ async-http-client ---
>>>> [INFO]
>>>> [INFO] ---
>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>> (enforce-versions) @ async-http-client ---
>>>> [INFO]
>>>> [INFO] ---
>>>> maven-resources-plugin:2.4.3:resources
>>>> (default-resources) @ async-http-client ---
>>>> [INFO] Using 'UTF-8' encoding to copy filtered
>>>> resources.
>>>> [INFO] skip non existing resourceDirectory
>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>> [INFO]
>>>> [INFO] --- maven-compiler-plugin:2.3.2:compile
>>>> (default-compile) @ async-http-client ---
>>>> [INFO] Compiling 158 source files to
>>>> /home/slorber/Bureau/ahc2/target/classes
>>>> [INFO]
>>>> *[INFO] ---
>>>> animal-sniffer-maven-plugin:1.6:check
>>>> (check-java-1.5-compat) @ async-http-client ---*
>>>> *[INFO] Checking unresolved references to
>>>> org.codehaus.mojo.signature:java15:1.0*
>>>> *[ERROR] Undefined reference:
>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V
>>>> in
>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class*
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 8.747s
>>>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>>>> [INFO] Final Memory: 30M/453M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> *[ERROR] Failed to execute goal
>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>> (check-java-1.5-compat) on project
>>>> async-http-client: Signature errors found.
>>>> Verify them and put @IgnoreJRERequirement on
>>>> them. -> [Help 1]*
>>>> [ERROR]
>>>> [ERROR] To see the full stack trace of the
>>>> errors, re-run Maven with the -e switch.
>>>> [ERROR] Re-run Maven using the -X switch to
>>>> enable full debug logging.
>>>> [ERROR]
>>>> [ERROR] For more information about the errors
>>>> and possible solutions, please read the
>>>> following articles:
>>>> [ERROR] [Help 1]
>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>
>>>>
>>>>
>>>> 2013/9/10 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>> Ok thank you, I'll try to implement that
>>>> today and will give you my feedback :)
>>>>
>>>>
>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>
>>>> Okay,
>>>>
>>>> I've committed my initial changes to
>>>> the AHC repository. Here's a summary
>>>> of the changes:
>>>>
>>>>
>>>> *Improvements to the
>>>> FeedableBodyGenerator (Grizzly's).
>>>> - Don't allow queueing of data before
>>>> initiateAsyncTransfer has been invoked.
>>>> In low memory
>>>> heaps, this could lead to an OOM if the
>>>> source is feeding too fast. The new
>>>> behavior is to
>>>> block until initiateAsyncTransfer is
>>>> called, at which time the blocked
>>>> thread may proceed with
>>>> the feed operation.
>>>> - Introduce the concept of a Feeder.
>>>> Implementations are responsible, at a
>>>> high level, for:
>>>> + letting the provider know that data
>>>> is available to be fed without blocking
>>>> + allowing the registration of a
>>>> callback that the Feeder implementation
>>>> may invoke
>>>> to signal that more data is available,
>>>> if it wasn't available at a previous
>>>> point in time.
>>>> - When using a Feeder with a secure
>>>> request, the SSL handshake will be
>>>> kicked off by the
>>>> initiateAsyncTransfer call, but feeding
>>>> of data will not occur until the
>>>> handshake is complete.
>>>> This is necessary as the SSLFilter will
>>>> queue up all writes until the handshake
>>>> is complete,
>>>> and currently, the buffer isn't tied in
>>>> with the transport flow control mechanism.
>>>> NOTE: This new SSL behavior is not
>>>> currently applied when invoking the
>>>> feed() method
>>>> outside the context of a Feeder. Still
>>>> need to address that.
>>>> - Exposed configuration of the async
>>>> write queue limit through the
>>>> FeedableBodyGenerator.
>>>> This is an improvement on using a
>>>> TransportCustomizer as any
>>>> configuration there is
>>>> transport-wide, and therefor applied to
>>>> all Connections. By exposing it here,
>>>> each feeder
>>>> may have a different byte limit.
>>>> - Improved documentation for this class*
>>>>
>>>>
>>>>
>>>> I recommend reading through the javadoc
>>>> comments in the source [1] for
>>>> FeedableBodyGenerator (comments welcome).
>>>> Additionally, I would re-work your code
>>>> to leverage the Feeder instead of
>>>> calling feed() directly.
>>>>
>>>> If you have issues implementing Feeder,
>>>> do let us know.
>>>>
>>>> If you have additional questions,
>>>> again, let us know.
>>>>
>>>> Thanks,
>>>> -rl
>>>>
>>>> [1]
>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>
>>>>
>>>>
>>>> Ryan Lubke wrote:
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>> So in the end I've end up with an
>>>>>> implementation that's working for me.
>>>>>>
>>>>>>
>>>>>> I think there are 2 bugs:
>>>>>>
>>>>>> 1) The bytes can accumulate in the
>>>>>> FeedableBodyGenerator queue if the
>>>>>> initialize(ctx) method is not called
>>>>>> fast enough.
>>>>>> This can be solved by using a
>>>>>> BlockingQueue of size 1 and the put()
>>>>>> method.
>>>>>>
>>>>>> 2) Once the context is injected, the
>>>>>> FeedableBodyGenerator flushes the queue.
>>>>>> The matter is that if the connection
>>>>>> is new, not warmed up by a previous
>>>>>> request, then the SSL handshake is
>>>>>> not done yet, and it seems that the
>>>>>> bytes are accumulated in some part of
>>>>>> the SSL filter which doesn't deliver
>>>>>> them to the connection until the
>>>>>> handshake has completed,
>>>>>> so c.canWrite() continues to return true.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have replaced some part of the
>>>>>> FeedableBodyGenerator to test this
>>>>>> and it works pretty fine. See what I
>>>>>> have changed:
>>>>>>
>>>>>> 1)
>>>>>> private final
>>>>>> BlockingQueue<BodyPart> queue = new
>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>
>>>>>>
>>>>>> 2)
>>>>>> public void feed(final Buffer
>>>>>> buffer, final boolean last) throws
>>>>>> IOException {
>>>>>> try {
>>>>>> queue.put(new
>>>>>> BodyPart(buffer, last));
>>>>>> } catch (InterruptedException
>>>>>> e) {
>>>>>> throw new RuntimeException(e);
>>>>>> }
>>>>>> queueSize.incrementAndGet();
>>>>>> if (context != null) {
>>>>>>
>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>> flushQueue(true);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> private void
>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>>>> fcc) {
>>>>>> if (
>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>> while (
>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>> try { Thread.sleep(10); }
>>>>>> catch ( Exception e ) { throw new
>>>>>> RuntimeException(e); }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> private boolean
>>>>>> connectionIsReadyToWrite(FilterChainContext
>>>>>> fcc) {
>>>>>> Connection connection =
>>>>>> fcc.getConnection();
>>>>>> SSLEngine sslEngine =
>>>>>> SSLUtils.getSSLEngine(connection);
>>>>>> return sslEngine != null &&
>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think?
>>>>>
>>>>> We had come to similar conclusions on
>>>>> this end. I'm still working through
>>>>> testing the idea I mentioned
>>>>> previously (took longer than I
>>>>> expected - sorry).
>>>>> I hope to have something for you to
>>>>> test very soon.
>>>>>
>>>>> Note that it will be taking the above
>>>>> into account as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/5 Sébastien Lorber
>>>>>> <lorber.sebastien_at_gmail.com
>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>
>>>>>>
>>>>>> I have tried to put a while (
>>>>>> context == null ) Thread.sleep
>>>>>> but it doesn't seem to work, when
>>>>>> the context gets injected, after
>>>>>> the sleeps, there's an OOM
>>>>>>
>>>>>> So I hope you'll have more
>>>>>> success with your alternative :)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have done another test,
>>>>>> remember my code that worked,
>>>>>> which previously "warmed" the
>>>>>> Thread with an useless request.
>>>>>>
>>>>>> private Runnable
>>>>>> uploadSingleDocumentRunnable =
>>>>>> new Runnable() {
>>>>>> @Override
>>>>>> public void run() {
>>>>>> try {
>>>>>> getUselessSessionCode();
>>>>>> Thread.sleep(X);
>>>>>> uploadSingleDocument();
>>>>>> } catch ( Exception e ) {
>>>>>> throw new
>>>>>> RuntimeException("file upload
>>>>>> failed",e);
>>>>>> }
>>>>>> }
>>>>>> };
>>>>>>
>>>>>> I have put a sleep of X between
>>>>>> the useless warmup request, and
>>>>>> the real upload request
>>>>>>
>>>>>>
>>>>>> What I noticed is that there is a
>>>>>> very different behavior according
>>>>>> to the value of X
>>>>>>
>>>>>>
>>>>>> Under 10 seconds, it seems the
>>>>>> stuff is still warm, I can upload
>>>>>> the documents.
>>>>>> Around 10 seconds I get a stack
>>>>>> which seems to be "connection
>>>>>> closed" or something
>>>>>> Above 10 seconds, I get OOM like
>>>>>> if the stuff wasn't warm.
>>>>>>
>>>>>>
>>>>>> The stacks I get for 10 seconds
>>>>>> looks like
>>>>>>
>>>>>> Caused by:
>>>>>> javax.net.ssl.SSLException:
>>>>>> SSLEngine is CLOSED
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>
>>>>>> I think I got some other
>>>>>> different stacks saying
>>>>>> Connection Closed Remotely or
>>>>>> something like that.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So it seems that something is
>>>>>> bound to my thread, and it stays
>>>>>> bound to it for about 10 seconds,
>>>>>> do you have any idea what it
>>>>>> could be?
>>>>>> (My connection timeout setting
>>>>>> seems to have no effect on this
>>>>>> 10s threshold)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/5 Ryan Lubke
>>>>>> <ryan.lubke_at_oracle.com
>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>
>>>>>> That is one solution. I'm
>>>>>> working out an alternative
>>>>>> right now. Stay tuned!
>>>>>>
>>>>>>>
>>>>>>> Anyway it's not a problem, I
>>>>>>> think the
>>>>>>> FeedableBodyGenerator.feed()
>>>>>>> method just has to block
>>>>>>> until a context has been
>>>>>>> (and ThreadCache
>>>>>>> initialized) to avoid OOM errors
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>>
>>>>>>> What is very strange is
>>>>>>> that I tested
>>>>>>> with/without the same
>>>>>>> sessionCode with our
>>>>>>> previous code, the one
>>>>>>> not using
>>>>>>> FeedableBodyGenerator,
>>>>>>> which has a high memory
>>>>>>> consumption.
>>>>>>> Despites the fact it had
>>>>>>> high memory consumption,
>>>>>>> it seems work fine to
>>>>>>> upload multiple
>>>>>>> documents if allocated
>>>>>>> with a large heap, and
>>>>>>> the sessionCode seems to
>>>>>>> have no effect.
>>>>>>>
>>>>>>> On the new impl using
>>>>>>> the
>>>>>>> FeedableBodyGenerator,
>>>>>>> the sessionCode sent as
>>>>>>> a multipart bodypart
>>>>>>> seems to have an effect.
>>>>>>>
>>>>>>> I have tried to feed the
>>>>>>> queue before sending the
>>>>>>> request to AHC, but this
>>>>>>> leads to this exception
>>>>>>> (with/without
>>>>>>> sessionCode switching)
>>>>>>> Caused by:
>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>> Timeout exceeded
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>> at
>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien
>>>>>>> Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>> By the way, by using
>>>>>>> a low timeout with
>>>>>>> the same
>>>>>>> sessioncode, I got
>>>>>>> the following NPE:
>>>>>>>
>>>>>>> Caused by:
>>>>>>> java.lang.NullPointerException
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>
>>>>>>>
>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>> httpCtx =
>>>>>>>
>>>>>>> getHttpTransactionContext(c);
>>>>>>>
>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>
>>>>>>> I guess the httpCtx
>>>>>>> is not already
>>>>>>> available to be aborted
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien
>>>>>>> Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is right,
>>>>>>> here's a log I
>>>>>>> have when I use
>>>>>>> the same session
>>>>>>> code, ie the
>>>>>>> remote host is
>>>>>>> blocking the
>>>>>>> data or something.
>>>>>>> This is obtained
>>>>>>> by running 5
>>>>>>> parallel uploads.
>>>>>>>
>>>>>>> *Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> *Flushing queue
>>>>>>> of size 97 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> *Flushing queue
>>>>>>> of size 100 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> *Flushing queue
>>>>>>> of size 160 with
>>>>>>> allowBlocking =
>>>>>>> true*
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> java.lang.OutOfMemoryError:
>>>>>>> GC overhead
>>>>>>> limit exceeded
>>>>>>> Dumping heap to
>>>>>>> /ome/lorber/ureau/om
>>>>>>> ...
>>>>>>> Unable to create
>>>>>>> /ome/lorber/ureau/om:
>>>>>>> Le fichier existe
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Disconnected
>>>>>>> from the target
>>>>>>> VM, address:
>>>>>>> '127.0.0.1:49268
>>>>>>> <http://127.0.0.1:49268>',
>>>>>>> transport: 'socket'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Otherwise, with
>>>>>>> different
>>>>>>> session codes, I
>>>>>>> get the following:
>>>>>>>
>>>>>>> *Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> *Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> *Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> *Flushing queue
>>>>>>> of size 0 with
>>>>>>> allowBlocking =
>>>>>>> false*
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> Flushing queue
>>>>>>> of size 1 with
>>>>>>> allowBlocking = true
>>>>>>> ... and this
>>>>>>> continues
>>>>>>> without OOM
>>>>>>>
>>>>>>>
>>>>>>> So, this seems
>>>>>>> to be the problem.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think it would
>>>>>>> be great to be
>>>>>>> able to be able
>>>>>>> to choose the
>>>>>>> queue impl
>>>>>>> behind
>>>>>>> that FeedableBodyGenerator,
>>>>>>> like I suggested
>>>>>>> in my pull request.
>>>>>>>
>>>>>>> See here:
>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>
>>>>>>> Using a
>>>>>>> LinkedBlockingQueue
>>>>>>> seems to be a
>>>>>>> nice idea in
>>>>>>> this context,
>>>>>>> and in my case I
>>>>>>> would probably
>>>>>>> use a queue of
>>>>>>> size 1
>>>>>>>
>>>>>>>
>>>>>>> This would
>>>>>>> handle the
>>>>>>> blocking of the
>>>>>>> feed method,
>>>>>>> without having
>>>>>>> to use this:
>>>>>>> if
>>>>>>> (context != null) {
>>>>>>>
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Or perhaps the
>>>>>>> feed() method
>>>>>>> have to wait
>>>>>>> until a context
>>>>>>> is set in the
>>>>>>> BodyGenerator ?
>>>>>>>
>>>>>>>
>>>>>>> I think it would
>>>>>>> be more clear if
>>>>>>> the initializeAsynchronousTransfer
>>>>>>> simply didn't
>>>>>>> flush the queue
>>>>>>> but just setup
>>>>>>> the context.
>>>>>>> Then the feed
>>>>>>> method would
>>>>>>> block until
>>>>>>> there's a
>>>>>>> context set, and
>>>>>>> then flush the
>>>>>>> queue with
>>>>>>> blocking behavior.
>>>>>>>
>>>>>>> This is probably
>>>>>>> the next step,
>>>>>>> but as we are
>>>>>>> using AHC for
>>>>>>> async, it would
>>>>>>> probably be
>>>>>>> great if that
>>>>>>> blocking feed()
>>>>>>> method was
>>>>>>> called in a
>>>>>>> worker thread
>>>>>>> instead of our
>>>>>>> main thread.
>>>>>>> I won't use this
>>>>>>> but someone who
>>>>>>> really wants a
>>>>>>> non-blocking
>>>>>>> impl of
>>>>>>> performant
>>>>>>> multipart
>>>>>>> fileupload would
>>>>>>> probably need
>>>>>>> this, or will
>>>>>>> use an
>>>>>>> ExecutorService
>>>>>>> for the feeding
>>>>>>> operations as a
>>>>>>> workaround.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks again for
>>>>>>> your reactivity
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/4 Ryan
>>>>>>> Lubke
>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sébastien
>>>>>>> Lorber wrote:
>>>>>>>> I've
>>>>>>>> integrated
>>>>>>>> this change
>>>>>>>> and it
>>>>>>>> works fine
>>>>>>>> except a
>>>>>>>> little detail.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm
>>>>>>>> uploading
>>>>>>>> files to a
>>>>>>>> third party
>>>>>>>> API (a bit
>>>>>>>> like S3).
>>>>>>>> This API
>>>>>>>> requires a
>>>>>>>> "sessionCode"
>>>>>>>> in each
>>>>>>>> request. So
>>>>>>>> there is a
>>>>>>>> multipart
>>>>>>>> StringPart
>>>>>>>> with that
>>>>>>>> SessionCode.
>>>>>>>>
>>>>>>>> We used to
>>>>>>>> have a
>>>>>>>> cache which
>>>>>>>> holds the
>>>>>>>> sessionCode
>>>>>>>> 30min per
>>>>>>>> user so
>>>>>>>> that we do
>>>>>>>> not need to
>>>>>>>> init a new
>>>>>>>> session
>>>>>>>> each time.
>>>>>>>>
>>>>>>>> I had
>>>>>>>> troubles in
>>>>>>>> this very
>>>>>>>> specific
>>>>>>>> case: when
>>>>>>>> I upload 5
>>>>>>>> docs with
>>>>>>>> the same
>>>>>>>> session code.
>>>>>>>> When I
>>>>>>>> remove the
>>>>>>>> cache and
>>>>>>>> use 5
>>>>>>>> different
>>>>>>>> session
>>>>>>>> codes, it
>>>>>>>> works fine.
>>>>>>>>
>>>>>>>> I guess the
>>>>>>>> remote
>>>>>>>> service is
>>>>>>>> blocking
>>>>>>>> concurrent
>>>>>>>> uploads
>>>>>>>> with the
>>>>>>>> same
>>>>>>>> session
>>>>>>>> code. I
>>>>>>>> don't know
>>>>>>>> at all.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Where I
>>>>>>>> want to go
>>>>>>>> is that I
>>>>>>>> wouldn't
>>>>>>>> have
>>>>>>>> expected
>>>>>>>> Grizzly to OOM
>>>>>>>>
>>>>>>>>
>>>>>>>> Avertissement:
>>>>>>>> Exception
>>>>>>>> during
>>>>>>>> FilterChain
>>>>>>>> execution
>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>> Java heap space
>>>>>>>> at
>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>> at
>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Caused by:
>>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>>> null
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> multipart.body.generator.feeder.buffer=100000
>>>>>>>> -> size of
>>>>>>>> each Buffer
>>>>>>>> sent to the
>>>>>>>> FeedableBodyGenerator
>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>> (I tried
>>>>>>>> other
>>>>>>>> settings,
>>>>>>>> including
>>>>>>>> AUTO_SIZE)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have
>>>>>>>> any idea
>>>>>>>> why is
>>>>>>>> there an
>>>>>>>> OOM with
>>>>>>>> these settings?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Perhaps it
>>>>>>>> is because
>>>>>>>> the feed()
>>>>>>>> method of
>>>>>>>> FeedableBodyGenerator
>>>>>>>> doesn't
>>>>>>>> block until
>>>>>>>> the context
>>>>>>>> is
>>>>>>>> initialized.
>>>>>>>> I guess
>>>>>>>> the initializeAsynchronousTransfer
>>>>>>>> is called
>>>>>>>> only once
>>>>>>>> the
>>>>>>>> connection
>>>>>>>> is
>>>>>>>> established, and
>>>>>>>> perhaps a
>>>>>>>> lot of
>>>>>>>> Buffer are
>>>>>>>> added to
>>>>>>>> the queue...
>>>>>>> Yes, it's
>>>>>>> invoked once
>>>>>>> the request
>>>>>>> has been
>>>>>>> dispatched,
>>>>>>> so if the
>>>>>>> generator is
>>>>>>> fed a lot
>>>>>>> before the
>>>>>>> request, I
>>>>>>> could see
>>>>>>> this happening.
>>>>>>> I'll see
>>>>>>> what I can
>>>>>>> do to
>>>>>>> alleviate
>>>>>>> that case.
>>>>>>>
>>>>>>>>
>>>>>>>> But I'm not
>>>>>>>> sure at all
>>>>>>>> because the
>>>>>>>> session
>>>>>>>> code is
>>>>>>>> transmitted
>>>>>>>> as a
>>>>>>>> BodyPart
>>>>>>>> and I get
>>>>>>>> the same
>>>>>>>> problem if
>>>>>>>> i put it as
>>>>>>>> the first
>>>>>>>> or last
>>>>>>>> multipart.
>>>>>>>>
>>>>>>>> It's not a
>>>>>>>> big deal,
>>>>>>>> perhaps I
>>>>>>>> should
>>>>>>>> always use
>>>>>>>> a different
>>>>>>>> session
>>>>>>>> code for
>>>>>>>> concurrent
>>>>>>>> operations
>>>>>>>> but I'd
>>>>>>>> like to be
>>>>>>>> sure that
>>>>>>>> we won't
>>>>>>>> have this
>>>>>>>> issue in
>>>>>>>> production...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/3
>>>>>>>> Ryan Lubke
>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>
>>>>>>>> Good
>>>>>>>> catch.
>>>>>>>> Fixed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien
>>>>>>>> Lorber
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There's a
>>>>>>>>> little
>>>>>>>>> mistake in
>>>>>>>>> the
>>>>>>>>> grizzly ahc
>>>>>>>>> provider
>>>>>>>>> relative
>>>>>>>>> to the
>>>>>>>>> write
>>>>>>>>> queue
>>>>>>>>> size.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>
>>>>>>>>> As you
>>>>>>>>> can
>>>>>>>>> see,
>>>>>>>>> the
>>>>>>>>> TransportCustomizer
>>>>>>>>> is
>>>>>>>>> called, and
>>>>>>>>> then
>>>>>>>>> the
>>>>>>>>> write
>>>>>>>>> queue
>>>>>>>>> size
>>>>>>>>> (among
>>>>>>>>> other
>>>>>>>>> things) is
>>>>>>>>> set to
>>>>>>>>> AUTO_SIZE
>>>>>>>>> (instead
>>>>>>>>> of
>>>>>>>>> previously
>>>>>>>>> UNLIMITED)
>>>>>>>>>
>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I
>>>>>>>>> think
>>>>>>>>> the
>>>>>>>>> default settings
>>>>>>>>> like
>>>>>>>>> this
>>>>>>>>> AUTO_SIZE
>>>>>>>>> attribute
>>>>>>>>> should
>>>>>>>>> be set
>>>>>>>>> before
>>>>>>>>> the
>>>>>>>>> customization
>>>>>>>>> of the
>>>>>>>>> transport,
>>>>>>>>> or
>>>>>>>>> they
>>>>>>>>> would
>>>>>>>>> override
>>>>>>>>> the
>>>>>>>>> value
>>>>>>>>> we
>>>>>>>>> customized.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This
>>>>>>>>> is
>>>>>>>>> actually
>>>>>>>>> my
>>>>>>>>> case,
>>>>>>>>> since
>>>>>>>>> I
>>>>>>>>> can't
>>>>>>>>> reproduce
>>>>>>>>> my
>>>>>>>>> "bug"
>>>>>>>>> which
>>>>>>>>> is
>>>>>>>>> "high
>>>>>>>>> memory
>>>>>>>>> consumption",
>>>>>>>>> even
>>>>>>>>> when
>>>>>>>>> using
>>>>>>>>> -1 /
>>>>>>>>> UNLIMITED
>>>>>>>>> in the
>>>>>>>>> TransportCustomizer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This
>>>>>>>>> could
>>>>>>>>> work
>>>>>>>>> fine
>>>>>>>>> for me
>>>>>>>>> with
>>>>>>>>> AUTO_SIZE,
>>>>>>>>> but
>>>>>>>>> I'd
>>>>>>>>> rather
>>>>>>>>> be
>>>>>>>>> able
>>>>>>>>> to
>>>>>>>>> tune
>>>>>>>>> this
>>>>>>>>> parameter
>>>>>>>>> during
>>>>>>>>> load
>>>>>>>>> tests
>>>>>>>>> to see
>>>>>>>>> the
>>>>>>>>> effect.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/8/31
>>>>>>>>> Sebastien
>>>>>>>>> Lorber
>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> i
>>>>>>>>> will
>>>>>>>>> ckeck
>>>>>>>>> that
>>>>>>>>> on
>>>>>>>>> monday.
>>>>>>>>> I
>>>>>>>>> can now
>>>>>>>>> upload
>>>>>>>>> a
>>>>>>>>> 500m
>>>>>>>>> file
>>>>>>>>> with
>>>>>>>>> 40m heap
>>>>>>>>> size
>>>>>>>>> ;)
>>>>>>>>>
>>>>>>>>> Envoyé
>>>>>>>>> de
>>>>>>>>> mon iPhone
>>>>>>>>>
>>>>>>>>> Le
>>>>>>>>> 30
>>>>>>>>> août
>>>>>>>>> 2013
>>>>>>>>> à
>>>>>>>>> 20:49,
>>>>>>>>> Ryan
>>>>>>>>> Lubke
>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> I'm
>>>>>>>>>> going
>>>>>>>>>> to be
>>>>>>>>>> updating
>>>>>>>>>> the
>>>>>>>>>> Grizzly
>>>>>>>>>> provider
>>>>>>>>>> such
>>>>>>>>>> that
>>>>>>>>>> AUTO_SIZE
>>>>>>>>>> (not
>>>>>>>>>> AUTO_TUNE)
>>>>>>>>>> is the
>>>>>>>>>> default,
>>>>>>>>>> so you
>>>>>>>>>> can
>>>>>>>>>> avoid
>>>>>>>>>> the
>>>>>>>>>> use
>>>>>>>>>> of the
>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>
>>>>>>>>>> Ryan
>>>>>>>>>> Lubke
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Regarding
>>>>>>>>>>> your
>>>>>>>>>>> tuning
>>>>>>>>>>> question,
>>>>>>>>>>> I would
>>>>>>>>>>> probably
>>>>>>>>>>> set
>>>>>>>>>>> the
>>>>>>>>>>> value
>>>>>>>>>>> to
>>>>>>>>>>> AsyncQueueWriter.AUTO_TUNE
>>>>>>>>>>> (this
>>>>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>> four
>>>>>>>>>>> times
>>>>>>>>>>> the
>>>>>>>>>>> socket
>>>>>>>>>>> write
>>>>>>>>>>> buffer)
>>>>>>>>>>> and
>>>>>>>>>>> see
>>>>>>>>>>> how
>>>>>>>>>>> that
>>>>>>>>>>> works.
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> A
>>>>>>>>>>>> question
>>>>>>>>>>>> first.
>>>>>>>>>>>> With
>>>>>>>>>>>> these
>>>>>>>>>>>> changes,
>>>>>>>>>>>> your
>>>>>>>>>>>> memory
>>>>>>>>>>>> usage
>>>>>>>>>>>> is
>>>>>>>>>>>> more
>>>>>>>>>>>> inline
>>>>>>>>>>>> with
>>>>>>>>>>>> what
>>>>>>>>>>>> you
>>>>>>>>>>>> were
>>>>>>>>>>>> looking
>>>>>>>>>>>> for?
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien
>>>>>>>>>>>> Lorber
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> By
>>>>>>>>>>>>> the
>>>>>>>>>>>>> way,
>>>>>>>>>>>>> do
>>>>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>> any
>>>>>>>>>>>>> idea
>>>>>>>>>>>>> when
>>>>>>>>>>>>> the
>>>>>>>>>>>>> 1.7.20
>>>>>>>>>>>>> will
>>>>>>>>>>>>> be
>>>>>>>>>>>>> released
>>>>>>>>>>>>> (with
>>>>>>>>>>>>> these
>>>>>>>>>>>>> new
>>>>>>>>>>>>> improvements?)
>>>>>>>>>>>>>
>>>>>>>>>>>>> We
>>>>>>>>>>>>> would
>>>>>>>>>>>>> like
>>>>>>>>>>>>> to
>>>>>>>>>>>>> know
>>>>>>>>>>>>> if
>>>>>>>>>>>>> we
>>>>>>>>>>>>> wait
>>>>>>>>>>>>> for
>>>>>>>>>>>>> a release
>>>>>>>>>>>>> or
>>>>>>>>>>>>> if
>>>>>>>>>>>>> we
>>>>>>>>>>>>> install
>>>>>>>>>>>>> our
>>>>>>>>>>>>> own
>>>>>>>>>>>>> temp
>>>>>>>>>>>>> release
>>>>>>>>>>>>> on
>>>>>>>>>>>>> Nexus
>>>>>>>>>>>>> :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/30
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank
>>>>>>>>>>>>> you,
>>>>>>>>>>>>> it
>>>>>>>>>>>>> works
>>>>>>>>>>>>> fine!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I just
>>>>>>>>>>>>> had
>>>>>>>>>>>>> to
>>>>>>>>>>>>> modify
>>>>>>>>>>>>> a single
>>>>>>>>>>>>> line
>>>>>>>>>>>>> after
>>>>>>>>>>>>> your
>>>>>>>>>>>>> commit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>> ->
>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If
>>>>>>>>>>>>> I let
>>>>>>>>>>>>> the
>>>>>>>>>>>>> initial
>>>>>>>>>>>>> value
>>>>>>>>>>>>> (-1)
>>>>>>>>>>>>> it
>>>>>>>>>>>>> won't
>>>>>>>>>>>>> block,
>>>>>>>>>>>>> canWrite
>>>>>>>>>>>>> always
>>>>>>>>>>>>> returns
>>>>>>>>>>>>> true
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Btw,
>>>>>>>>>>>>> on
>>>>>>>>>>>>> AHC
>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>> find
>>>>>>>>>>>>> any
>>>>>>>>>>>>> way
>>>>>>>>>>>>> to
>>>>>>>>>>>>> pass
>>>>>>>>>>>>> this
>>>>>>>>>>>>> value
>>>>>>>>>>>>> as
>>>>>>>>>>>>> a config
>>>>>>>>>>>>> attribute,
>>>>>>>>>>>>> neither
>>>>>>>>>>>>> the
>>>>>>>>>>>>> size
>>>>>>>>>>>>> of
>>>>>>>>>>>>> the
>>>>>>>>>>>>> write
>>>>>>>>>>>>> buffer
>>>>>>>>>>>>> you
>>>>>>>>>>>>> talked
>>>>>>>>>>>>> about.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> end,
>>>>>>>>>>>>> is
>>>>>>>>>>>>> there
>>>>>>>>>>>>> a way
>>>>>>>>>>>>> with
>>>>>>>>>>>>> current
>>>>>>>>>>>>> AHC
>>>>>>>>>>>>> code
>>>>>>>>>>>>> to
>>>>>>>>>>>>> use
>>>>>>>>>>>>> this
>>>>>>>>>>>>> "canWrite
>>>>>>>>>>>>> = false"
>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If
>>>>>>>>>>>>> not,
>>>>>>>>>>>>> can
>>>>>>>>>>>>> you
>>>>>>>>>>>>> please
>>>>>>>>>>>>> provide
>>>>>>>>>>>>> a way
>>>>>>>>>>>>> to
>>>>>>>>>>>>> set
>>>>>>>>>>>>> this
>>>>>>>>>>>>> behavior
>>>>>>>>>>>>> on
>>>>>>>>>>>>> v1.7.20
>>>>>>>>>>>>> ? thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> PS:
>>>>>>>>>>>>> does
>>>>>>>>>>>>> it
>>>>>>>>>>>>> make
>>>>>>>>>>>>> sens
>>>>>>>>>>>>> to
>>>>>>>>>>>>> use
>>>>>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>> number
>>>>>>>>>>>>> of
>>>>>>>>>>>>> bytes
>>>>>>>>>>>>> un
>>>>>>>>>>>>> the
>>>>>>>>>>>>> feed(Buffer)
>>>>>>>>>>>>> method
>>>>>>>>>>>>> and
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>> ? do
>>>>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>> any
>>>>>>>>>>>>> tuning
>>>>>>>>>>>>> recommandation?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please
>>>>>>>>>>>>> disregard.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>> a sample
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>> performing
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> feed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>> recommend
>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> Connection.canWrite()
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener)
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>> default,
>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> four
>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>> buffer).
>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>> exceeds
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> value,
>>>>>>>>>>>>>>> canWrite()
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>> false.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> occurs,
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>> a WriteListener
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> notified
>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> configured
>>>>>>>>>>>>>>> max
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>> simulate
>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>> callback
>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>> invoked.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>> FutureImpl<Boolean>
>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>> = Futures.createSafeFuture();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> //
>>>>>>>>>>>>>>> Connection
>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> obtained
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> calling
>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> connection.notifyCanWrite(new
>>>>>>>>>>>>>>> WriteHandler()
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>> throws
>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>> onError(Throwable
>>>>>>>>>>>>>>> t)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>> writeTimeout
>>>>>>>>>>>>>>> = 30;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>> (ExecutionException
>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>> (Exception
>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>> did
>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> a blocking
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> totally
>>>>>>>>>>>>>>>> useless
>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>> consuming
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>> blocks
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> feeding,
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>> intention
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> beginning.
>>>>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>> strategy
>>>>>>>>>>>>>>>> used?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I use
>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> use SameThreadIOStrategy
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>> a 70m
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> 50m,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> put
>>>>>>>>>>>>>>>> a Thread.sleep(30)
>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> I know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>> things
>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> send.
>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> possible?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> 100mo
>>>>>>>>>>>>>>>> heap,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>> a sleep
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> 100ms
>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>>> chance
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> MessageCloner
>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>> filter?
>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> cloned,
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> I remove
>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>> How?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>> trying
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>> a 500m
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>> cases
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> consumption
>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> x parallel
>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> according
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>> activity.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>> reduced
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> footprint
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>> allocation
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> concerned,
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> untunable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That
>>>>>>>>>>>>>>>> said,
>>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>> free
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>> a feature
>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>> size.
>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>> application
>>>>>>>>>>>>>>>> specific.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>> curious,
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> sending?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> buffers
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> a size
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 33842
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> limit
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> near
>>>>>>>>>>>>>>>>> half
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> capacity.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> tune
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> consumes
>>>>>>>>>>>>>>>>> less
>>>>>>>>>>>>>>>>> memory?
>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> ideal
>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> reviewing
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>> today,
>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>> again!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regarding
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> OOM:
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> stands
>>>>>>>>>>>>>>>>> now,
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> SSLFilter,
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>> a buffer
>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> order
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> accommodate
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> encrypted
>>>>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Depending
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>> endpoints,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> becoming
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> situation,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> pretty
>>>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>>>> shielded
>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>> I will
>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> leverage
>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> With
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> usecase
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> works,
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> uploaded
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>> a big
>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> body
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> that?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>> memory,
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>>> of:
>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> SSLUtils$3
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> BufferBuffers
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> WriteResult
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> exemple
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>> stacktrace:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>>>>> idea?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Excellent!
>>>>>>>>>>>>>>>>>> Looking
>>>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>> request!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> works
>>>>>>>>>>>>>>>>>>> fine,
>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>> a pull
>>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>> tomorrow
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> a better
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>> Part
>>>>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I created
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> OutputStream
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> redirects
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>>>> feeder.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>> I currently
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> feeder
>>>>>>>>>>>>>>>>>>> feeds
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>>> polling
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>> I need
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> expose
>>>>>>>>>>>>>>>>>>> a limit
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> something,
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> a thread
>>>>>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> filepart
>>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes,
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>> I was
>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> tackle
>>>>>>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>>>>> up
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ok
>>>>>>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think
>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>> do,
>>>>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>> bodyGenerator
>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Request
>>>>>>>>>>>>>>>>>>>> uploadRequest1
>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response>
>>>>>>>>>>>>>>>>>>>> asyncRes
>>>>>>>>>>>>>>>>>>>> = asyncHttpClient
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Response
>>>>>>>>>>>>>>>>>>>> uploadResponse
>>>>>>>>>>>>>>>>>>>> = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I guess
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> interesting
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/26
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> / Grizzly
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> streaming,
>>>>>>>>>>>>>>>>>>>> I mean
>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>>>> a byte[]
>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>> contans
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>> file,
>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>> a large
>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>>>> tryied
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> a Heap
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> ByteBuffer
>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> managers,
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> reallocate=true/false
>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>> success.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>> content
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> appended
>>>>>>>>>>>>>>>>>>>> wto
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> BufferOutputStream,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> underlying
>>>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>> uploading
>>>>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> Grizzly?
>>>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>>>>>>> tied
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> Multipart
>>>>>>>>>>>>>>>>>>>> uploads,
>>>>>>>>>>>>>>>>>>>> you'd
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> convert
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> multipart
>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> Buffers
>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>> a closer
>>>>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> area
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Btw
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>> upload.
>>>>>>>>>>>>>>>>>>>> I receive
>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> transmit
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> a storage
>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>> (like
>>>>>>>>>>>>>>>>>>>> S3).
>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> bevause
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> streaming
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>>>>> fle
>>>>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>> system,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> provides
>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> input
>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
>