Would it be possible to share you testing framework?
Sébastien Lorber wrote:
> Thanks,
>
>
>
> So in the end it seems I can run concurrent uploads. But it is kind of
> limited, with 10 threads it works fine, and with 20 threads it seems
> the flush call is sometimes done too early because the feed method is
> not blocking, which produces OOM in some cases. The threshold seems to
> be around 15.
>
>
> I found a solution which seems to work in every situation:
> - Use a worker thread pool > number of concurrent requests
> - Wait a bit before starting the feeding.
>
>
>
> My waiting code is:
>
> private volatile boolean alreadyFlushed = false;
> @Override
> public synchronized void flush() {
> if ( alreadyFlushed ) {
> return;
> }
> try { Thread.sleep(1000); } catch ( Exception e ) {
> e.printStackTrace(); }
> startFeeding();
> alreadyFlushed = true;
> }
>
>
>
> With a 35 worker thread pool and this waiting code, I can
> run**30 concurrent uploads of large files, without having performed
> any request before.
>
>
>
> If I remove any of these 2 tricks, I get problems.
>
>
>
> *1) If I put a low worker thread number:*
> I mean, for 30 concurrent uploads, it starts to fail even for
> something like 31/32 worker threads
>
> Sometime I get timeout stacks:
>
> Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
> at
> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
> at
> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
> at
> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
>
> Sometime I get OOM stacks:
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
> at
> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
> at org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
> at
> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
> at
> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
> at
> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
> at org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
> at
> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
> at org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
> at
> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
> at
> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
> at
> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
> at
> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
> at
> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
> at
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
> at
> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>
>
> Another very strange behavior I have is that some FileInputStream that
> are used for the requests are read prematurely for a reason I can't
> understand.
> Not all, but some of the 30 concurrent uploads. I can see just after
> the firing of the 30 upload requests in my log:
>
> [Grizzly(1)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_61187ada after having read 10632755 bytes
> [Grizzly(15)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_1ba83dc after having read 10632755 bytes
> [Grizzly(13)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_7c26e166 after having read 10632755 bytes
> [Grizzly(10)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_5c982f37 after having read 10632755 bytes
> [Grizzly(6)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_b43f35f after having read 10632755 bytes
> [Grizzly(4)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_1a1ee7c0 after having read 10632755 bytes
> [Grizzly(9)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_6300fba5 after having read 10632755 bytes
> [Grizzly(16)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_49d88b78 after having read 10632755 bytes
> [Grizzly(4)] INFO com....DocumentService$1:276 - Will close file
> stream java.io.FileInputStream_at_3b7b65f8 after having read 598016 bytes
> --asept. 12, 2013 5:54:14 PM
> org.glassfish.grizzly.filterchain.DefaultFilterChain execute
> Avertissement: Exception during FilterChain execution
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> This close operation on the file stream is supposed to happen after
> more than 30 seconds because the connection is slow and the file is 10M
>
> This generally leads to OOM errors, it seems the whole file is mounted
> in memory for unknown reason in some cases, but only when there are
> not enough worker threads.
>
>
> It's not a big deal, I guess it is "normal" to use at least 30 threads
> to run 30 concurrent uploads :) It's just that when this happens it's
> not easy to understand why we have OOMs and how it relates to the AHC
> grizzly thread pool.
>
>
>
> *2) If I remove that sleep(1000) before feeding or use a lower number*
> This leads to the same kind of behavior, with the InputStream being
> read prematurely.
>
>
>
>
> So I guess, the problem is that when the flush() method is called it
> seems the connection always return canWrite=true, so the blocking
> never happen until something happen and the connection starts to block
> the feeding
>
> I tried to debug that and here's the connection queue state:
>
> * connectionQueue = {org.glassfish.grizzly.asyncqueue.TaskQueue_at_4804}
> * isClosed = true
> * queue = {java.util.concurrent.ConcurrentLinkedQueue_at_5106} size = 0
> * currentElement =
> {java.util.concurrent.atomic.AtomicReference_at_5107}"null"
> * spaceInBytes =
> {java.util.concurrent.atomic.AtomicInteger_at_5108}"291331"
> * maxQueueSizeHolder = {org.glassfish.grizzly.nio.NIOConnection$1_at_5109}
> * writeHandlersCounter =
> {java.util.concurrent.atomic.AtomicInteger_at_5110}"0"
> * writeHandlersQueue =
> {java.util.concurrent.ConcurrentLinkedQueue_at_5111} size = 0
>
>
>
> Do you have any idea what could be the problem?
>
> This happens when flush() is called from both init() and
> handshakeComplete() methods
>
>
> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com
> <mailto:ryan.lubke_at_oracle.com>>
>
>
>
> Sébastien Lorber wrote:
>>
>>
>>
>> I noticed something strange.
>> On the FileInputStream I have, I've added a log on the close() of
>> the stream which is called once the whole file has been read to
>> be sent to the feeder.
>>
>> 1) If I perform a request (ie my session init request in the
>> previous discussions) before doing my multipart upload, the
>> thread that does execute the feeding is the thread that fires the
>> request, and not a Grizzly worker.
>> [Thread-30] INFO Will close file stream
>> java.io.FileInputStream_at_27fe4315 after having read 1440304 bytes
>>
>>
>> 2) If I don't do any request before firing the multipart upload,
>> the thread that does the feeding is a Grizzly threadpool worker
>> thread:
>> [Grizzly(22)] INFO Will close file stream
>> java.io.FileInputStream_at_59ac4002 after having read 1440304 bytes
>>
>>
>> Is this normal? I would expect a worker thread to always be used,
>> and the main applicative thread that performs the request to
>> never be blocking. (but it's not such an issue for me, we don't
>> have a reactive non-blocking app anyway)
> It's currently expected behavior. Will need to re-evaluate this
> based on the new semantics of the FeedableBodyGenerator.
>>
>>
>> On the 1st case, here's the stacktrace when the feed method is
>> called:
>>
>> *at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.initializeAsynchronousTransfer(FeedableBodyGenerator.java:178)*
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$BodyGeneratorBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2210)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:564)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:913)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:795)
>> at
>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>> at
>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:437)
>> at
>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:387)
>> at
>> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:361)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:307)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:224)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$1.completed(GrizzlyAsyncHttpProvider.java:210)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2289)
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:244)
>> *at
>> com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:534)*
>>
>> So it seems this happen when the handshake has already been done
>> when *initializeAsynchronousTransfer* is called, so that we do
>> not go through the HandshakeListener
>>
>>
>> Notice that this case seems to also happen in a few threads on
>> the 2nd case, but most of the threads are Grizzly workers.
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/9/12 Sébastien Lorber <lorber.sebastien_at_gmail.com
>> <mailto:lorber.sebastien_at_gmail.com>>
>>
>>
>>
>> Hi.
>>
>>
>> Thanks, it seems to work.
>>
>> I would suggest to throw IOException on the flush method
>> since the feed method is supposed to be called here
>>
>>
>> My implementation of flush is:
>>
>> @Override
>> public void flush() {
>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>> try ( OutputStream outputStream =
>> createFeedingOutputStream() ) {
>>
>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>> } catch (Exception e) {
>> throw new IllegalStateException("Unable to feed the
>> FeedableBodyGenerator",e);
>> }
>> }
>>
>> Is this correct? The OutputStream redirects the bytes written
>> to the feed(Buffer) method
>>
>>
>>
>>
>>
>>
>> There seems to be some concurrency issue. Because the upload
>> of 1 file seems fine, but when using multiple threads, I
>> often get the following stack:
>> Caused by: java.io.IOException: Stream Closed
>> at java.io.FileInputStream.readBytes(Native Method)
>> at java.io.FileInputStream.read(FileInputStream.java:242)
>> at
>> com.google.common.io.CountingInputStream.read(CountingInputStream.java:62)
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:178)
>> at com.ning.http.multipart.Part.send(Part.java:331)
>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>
>>
>> This is because the flush() method is called multiple times
>> for the same request on some cases.
>> I guess this is not supposed to happen.
>> What I understand is that the flush() method is supposed to
>> be called only once.
>>
>>
>> Using debug logging breakpoints I get the following:
>>
>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 0 started
>> myapp--api-test 12/09/2013-12:20:46.042 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 1 started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 2 started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 3 started
>> myapp--api-test 12/09/2013-12:20:46.043 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 4 started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 5 started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 6 started
>> myapp--api-test 12/09/2013-12:20:46.044 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 7 started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 8 started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 9 started
>> myapp--api-test 12/09/2013-12:20:46.045 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 10 started
>> myapp--api-test 12/09/2013-12:20:46.046 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 11 started
>> myapp--api-test 12/09/2013-12:20:46.047 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 12 started
>> myapp--api-test 12/09/2013-12:20:46.048 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 13 started
>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 14 started
>> myapp--api-test 12/09/2013-12:20:46.049 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 15 started
>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 16 started
>> myapp--api-test 12/09/2013-12:20:46.050 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 17 started
>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 18 started
>> myapp--api-test 12/09/2013-12:20:46.051 [] [] [main] INFO
>> com.myapp.perf.DocumentUploadPerfIntegrationTest:77 - Thread
>> 19 started
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_417de6ff
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1c940409
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3e888183
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_17840db8
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_2cbad94b
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_3d5d9ee8
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_51557949
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6f95de2f
>> Adding handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_7799b411
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_5befaa07*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>> *Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_102873a6*
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_346d784c
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_674735a8
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_64c0a4ae
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_480f9510
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_6c0d6ef7
>> Completing and removing handshake listener for
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator_at_1daf8fd8
>>
>>
>> As you can see the same HandshakeListener seems to be called
>> multiple times for the same FeedableBodyGenerator
>>
>> When this happens, the stack is:
>>
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$1.onComplete(FeedableBodyGenerator.java:198)
>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>> at
>> org.glassfish.grizzly.ssl.SSLBaseFilter.notifyHandshakeComplete(SSLBaseFilter.java:880)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:282)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>> at
>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>> at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>
>>
>>
>>
>> So I tried with the following code:
>>
>> private boolean alreadyFlushed = false;
>> @Override
>> public synchronized void flush() {
>> if ( alreadyFlushed ) {
>> return;
>> }
>> startFeeding();
>> alreadyFlushed = true;
>> }
>>
>> It works fine when upload small files.
>>
>> But with larger files, I often get TimeoutException stacks
>> for some of the threads:
>>
>> Caused by: java.util.concurrent.TimeoutException
>> at
>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>> at
>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.block(FeedableBodyGenerator.java:349)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.blockUntilQueueFree(FeedableBodyGenerator.java:339)
>> at
>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator$BaseFeeder.feed(FeedableBodyGenerator.java:306)
>>
>>
>>
>>
>> Did I miss something?
>>
>>
>>
>>
>>
>>
>> 2013/9/12 Ryan Lubke <ryan.lubke_at_oracle.com
>> <mailto:ryan.lubke_at_oracle.com>>
>>
>> Committed another small change. Please make sure you're
>> at the latest when you build.
>>
>> -rl
>>
>>
>> Ryan Lubke wrote:
>>> Okay, I've committed another set of refactorings to the
>>> FeedableBodyGenerator.
>>>
>>> For your use case, you should extend
>>> FeedableBodyGenerator.SimpleFeeder.
>>>
>>> Let me know if you run into issues.
>>>
>>>
>>>
>>> Sébastien Lorber wrote:
>>>> Yes I think it would, so that I can feed the queue at once
>>>>
>>>> One thread will be locked during the feeding for
>>>> nothing but it's not a real problem in my usecase.
>>>>
>>>>
>>>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>
>>>> Would having a different listener that will be
>>>> notified once async transferring has been started
>>>> work better for you?
>>>>
>>>> Something like:
>>>>
>>>> onAsyncTransferInitiated() {
>>>> // invoke your feed method
>>>> }
>>>>
>>>> ?
>>>>
>>>>
>>>>
>>>> Sébastien Lorber wrote:
>>>>> Unfortunatly I won't be able to use the Feeder
>>>>> non-blocking stuff for now, because of how the
>>>>> multipart request in handled in AHC
>>>>>
>>>>>
>>>>> Here's my feeding method:
>>>>>
>>>>> public void feed() throws IOException {
>>>>> Part[] partsArray = parts.toArray(new
>>>>> Part[parts.size()]);
>>>>> try ( OutputStream outputStream =
>>>>> createFeedingOutputStream() ) {
>>>>>
>>>>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>>>>> } catch (Exception e) {
>>>>> throw new IllegalStateException("Unable to
>>>>> feed the FeedableBodyGenerator",e);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> As you can see, the multipart Parts array can only
>>>>> be pushed to the OutputStream, I don't have any
>>>>> way to "pull" the data when the canFeed() method
>>>>> is triggered.
>>>>>
>>>>>
>>>>> But I've seen that there's
>>>>> a com.ning.http.multipart.MultipartBody#read that
>>>>> seems to provide a memory efficient way to pull
>>>>> data from a Multipart body...
>>>>>
>>>>> Should see what I come up with this
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> It seems the Feeder is highly recommended but
>>>>> not mandatory so I tried without.
>>>>>
>>>>> With my existing code it seems there is a
>>>>> synchronization problem.
>>>>>
>>>>>
>>>>> The feeding threads get locked to
>>>>> the prematureFeed.get();
>>>>>
>>>>> So the Grizzly kernel threads are unable to
>>>>> acquire the lock required to enter
>>>>> the initializeAsynchronousTransfer method
>>>>>
>>>>>
>>>>>
>>>>> Will try with an implementation of Feeder
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> Hmmm it seems I have a problem with one of
>>>>> your maven plugins. I'll try to bypass it,
>>>>> but for info:
>>>>>
>>>>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>>>>> [WARNING]
>>>>> [WARNING] Some problems were encountered
>>>>> while building the effective settings
>>>>> [WARNING]
>>>>> 'profiles.profile[default].repositories.repository.id
>>>>> <http://repositories.repository.id>' must
>>>>> be unique but found duplicate repository
>>>>> with id fullsix-maven-repository @
>>>>> /home/slorber/.m2/settings.xml
>>>>> [WARNING]
>>>>> [INFO] Scanning for projects...
>>>>> [INFO]
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Building Asynchronous Http Client
>>>>> 1.7.20-SNAPSHOT
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO]
>>>>> [INFO] --- maven-clean-plugin:2.4.1:clean
>>>>> (default-clean) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>>> (enforce-maven) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-enforcer-plugin:1.0-beta-1:enforce
>>>>> (enforce-versions) @ async-http-client ---
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-resources-plugin:2.4.3:resources
>>>>> (default-resources) @ async-http-client ---
>>>>> [INFO] Using 'UTF-8' encoding to copy
>>>>> filtered resources.
>>>>> [INFO] skip non existing resourceDirectory
>>>>> /home/slorber/Bureau/ahc2/src/main/resources
>>>>> [INFO]
>>>>> [INFO] ---
>>>>> maven-compiler-plugin:2.3.2:compile
>>>>> (default-compile) @ async-http-client ---
>>>>> [INFO] Compiling 158 source files to
>>>>> /home/slorber/Bureau/ahc2/target/classes
>>>>> [INFO]
>>>>> *[INFO] ---
>>>>> animal-sniffer-maven-plugin:1.6:check
>>>>> (check-java-1.5-compat) @
>>>>> async-http-client ---*
>>>>> *[INFO] Checking unresolved references to
>>>>> org.codehaus.mojo.signature:java15:1.0*
>>>>> *[ERROR] Undefined reference:
>>>>> java/io/IOException.<init>(Ljava/lang/Throwable;)V
>>>>> in
>>>>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class*
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 8.747s
>>>>> [INFO] Finished at: Tue Sep 10 11:25:41
>>>>> CEST 2013
>>>>> [INFO] Final Memory: 30M/453M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> *[ERROR] Failed to execute goal
>>>>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>>>>> (check-java-1.5-compat) on project
>>>>> async-http-client: Signature errors found.
>>>>> Verify them and put @IgnoreJRERequirement
>>>>> on them. -> [Help 1]*
>>>>> [ERROR]
>>>>> [ERROR] To see the full stack trace of the
>>>>> errors, re-run Maven with the -e switch.
>>>>> [ERROR] Re-run Maven using the -X switch
>>>>> to enable full debug logging.
>>>>> [ERROR]
>>>>> [ERROR] For more information about the
>>>>> errors and possible solutions, please read
>>>>> the following articles:
>>>>> [ERROR] [Help 1]
>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/10 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> Ok thank you, I'll try to implement
>>>>> that today and will give you my
>>>>> feedback :)
>>>>>
>>>>>
>>>>> 2013/9/10 Ryan Lubke
>>>>> <ryan.lubke_at_oracle.com
>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>
>>>>> Okay,
>>>>>
>>>>> I've committed my initial changes
>>>>> to the AHC repository. Here's a
>>>>> summary of the changes:
>>>>>
>>>>>
>>>>> *Improvements to the
>>>>> FeedableBodyGenerator (Grizzly's).
>>>>> - Don't allow queueing of data
>>>>> before initiateAsyncTransfer has
>>>>> been invoked. In low memory
>>>>> heaps, this could lead to an OOM
>>>>> if the source is feeding too fast.
>>>>> The new behavior is to
>>>>> block until initiateAsyncTransfer
>>>>> is called, at which time the
>>>>> blocked thread may proceed with
>>>>> the feed operation.
>>>>> - Introduce the concept of a
>>>>> Feeder. Implementations are
>>>>> responsible, at a high level, for:
>>>>> + letting the provider know that
>>>>> data is available to be fed
>>>>> without blocking
>>>>> + allowing the registration of a
>>>>> callback that the Feeder
>>>>> implementation may invoke
>>>>> to signal that more data is
>>>>> available, if it wasn't available
>>>>> at a previous point in time.
>>>>> - When using a Feeder with a
>>>>> secure request, the SSL handshake
>>>>> will be kicked off by the
>>>>> initiateAsyncTransfer call, but
>>>>> feeding of data will not occur
>>>>> until the handshake is complete.
>>>>> This is necessary as the SSLFilter
>>>>> will queue up all writes until the
>>>>> handshake is complete,
>>>>> and currently, the buffer isn't
>>>>> tied in with the transport flow
>>>>> control mechanism.
>>>>> NOTE: This new SSL behavior is not
>>>>> currently applied when invoking
>>>>> the feed() method
>>>>> outside the context of a Feeder.
>>>>> Still need to address that.
>>>>> - Exposed configuration of the
>>>>> async write queue limit through
>>>>> the FeedableBodyGenerator.
>>>>> This is an improvement on using a
>>>>> TransportCustomizer as any
>>>>> configuration there is
>>>>> transport-wide, and therefor
>>>>> applied to all Connections. By
>>>>> exposing it here, each feeder
>>>>> may have a different byte limit.
>>>>> - Improved documentation for this
>>>>> class*
>>>>>
>>>>>
>>>>>
>>>>> I recommend reading through the
>>>>> javadoc comments in the source [1]
>>>>> for FeedableBodyGenerator
>>>>> (comments welcome).
>>>>> Additionally, I would re-work your
>>>>> code to leverage the Feeder
>>>>> instead of calling feed() directly.
>>>>>
>>>>> If you have issues implementing
>>>>> Feeder, do let us know.
>>>>>
>>>>> If you have additional questions,
>>>>> again, let us know.
>>>>>
>>>>> Thanks,
>>>>> -rl
>>>>>
>>>>> [1]
>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>
>>>>>
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber wrote:
>>>>>>> So in the end I've end up with
>>>>>>> an implementation that's working
>>>>>>> for me.
>>>>>>>
>>>>>>>
>>>>>>> I think there are 2 bugs:
>>>>>>>
>>>>>>> 1) The bytes can accumulate in
>>>>>>> the FeedableBodyGenerator queue
>>>>>>> if the initialize(ctx) method is
>>>>>>> not called fast enough.
>>>>>>> This can be solved by using a
>>>>>>> BlockingQueue of size 1 and the
>>>>>>> put() method.
>>>>>>>
>>>>>>> 2) Once the context is injected,
>>>>>>> the FeedableBodyGenerator
>>>>>>> flushes the queue.
>>>>>>> The matter is that if the
>>>>>>> connection is new, not warmed up
>>>>>>> by a previous request, then the
>>>>>>> SSL handshake is not done yet,
>>>>>>> and it seems that the bytes are
>>>>>>> accumulated in some part of the
>>>>>>> SSL filter which doesn't deliver
>>>>>>> them to the connection until the
>>>>>>> handshake has completed,
>>>>>>> so c.canWrite() continues to
>>>>>>> return true.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have replaced some part of the
>>>>>>> FeedableBodyGenerator to test
>>>>>>> this and it works pretty fine.
>>>>>>> See what I have changed:
>>>>>>>
>>>>>>> 1)
>>>>>>> private final
>>>>>>> BlockingQueue<BodyPart> queue =
>>>>>>> new
>>>>>>> LinkedBlockingQueue<BodyPart>(1);
>>>>>>>
>>>>>>>
>>>>>>> 2)
>>>>>>> public void feed(final
>>>>>>> Buffer buffer, final boolean
>>>>>>> last) throws IOException {
>>>>>>> try {
>>>>>>> queue.put(new
>>>>>>> BodyPart(buffer, last));
>>>>>>> } catch
>>>>>>> (InterruptedException e) {
>>>>>>> throw new
>>>>>>> RuntimeException(e);
>>>>>>> }
>>>>>>> queueSize.incrementAndGet();
>>>>>>> if (context != null) {
>>>>>>>
>>>>>>> blockUntilConnectionIsReadyToWrite(context);
>>>>>>> flushQueue(true);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private void
>>>>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>>>>> fcc) {
>>>>>>> if (
>>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>>> while (
>>>>>>> !connectionIsReadyToWrite(fcc) ) {
>>>>>>> try {
>>>>>>> Thread.sleep(10); } catch (
>>>>>>> Exception e ) { throw new
>>>>>>> RuntimeException(e); }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private boolean
>>>>>>> connectionIsReadyToWrite(FilterChainContext
>>>>>>> fcc) {
>>>>>>> Connection connection =
>>>>>>> fcc.getConnection();
>>>>>>> SSLEngine sslEngine =
>>>>>>> SSLUtils.getSSLEngine(connection);
>>>>>>> return sslEngine != null
>>>>>>> &&
>>>>>>> !SSLUtils.isHandshaking(sslEngine);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think?
>>>>>>
>>>>>> We had come to similar
>>>>>> conclusions on this end. I'm
>>>>>> still working through testing the
>>>>>> idea I mentioned previously (took
>>>>>> longer than I expected - sorry).
>>>>>> I hope to have something for you
>>>>>> to test very soon.
>>>>>>
>>>>>> Note that it will be taking the
>>>>>> above into account as well.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Sébastien Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>>
>>>>>>> I have tried to put a while
>>>>>>> ( context == null )
>>>>>>> Thread.sleep but it doesn't
>>>>>>> seem to work, when the
>>>>>>> context gets injected, after
>>>>>>> the sleeps, there's an OOM
>>>>>>>
>>>>>>> So I hope you'll have more
>>>>>>> success with your alternative :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have done another test,
>>>>>>> remember my code that
>>>>>>> worked, which previously
>>>>>>> "warmed" the Thread with an
>>>>>>> useless request.
>>>>>>>
>>>>>>> private Runnable
>>>>>>> uploadSingleDocumentRunnable
>>>>>>> = new Runnable() {
>>>>>>> @Override
>>>>>>> public void run() {
>>>>>>> try {
>>>>>>> getUselessSessionCode();
>>>>>>> Thread.sleep(X);
>>>>>>> uploadSingleDocument();
>>>>>>> } catch ( Exception e ) {
>>>>>>> throw new
>>>>>>> RuntimeException("file
>>>>>>> upload failed",e);
>>>>>>> }
>>>>>>> }
>>>>>>> };
>>>>>>>
>>>>>>> I have put a sleep of X
>>>>>>> between the useless warmup
>>>>>>> request, and the real upload
>>>>>>> request
>>>>>>>
>>>>>>>
>>>>>>> What I noticed is that there
>>>>>>> is a very different behavior
>>>>>>> according to the value of X
>>>>>>>
>>>>>>>
>>>>>>> Under 10 seconds, it seems
>>>>>>> the stuff is still warm, I
>>>>>>> can upload the documents.
>>>>>>> Around 10 seconds I get a
>>>>>>> stack which seems to be
>>>>>>> "connection closed" or something
>>>>>>> Above 10 seconds, I get OOM
>>>>>>> like if the stuff wasn't warm.
>>>>>>>
>>>>>>>
>>>>>>> The stacks I get for 10
>>>>>>> seconds looks like
>>>>>>>
>>>>>>> Caused by:
>>>>>>> javax.net.ssl.SSLException:
>>>>>>> SSLEngine is CLOSED
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>> at
>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>>>>
>>>>>>> I think I got some other
>>>>>>> different stacks saying
>>>>>>> Connection Closed Remotely
>>>>>>> or something like that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So it seems that something
>>>>>>> is bound to my thread, and
>>>>>>> it stays bound to it for
>>>>>>> about 10 seconds, do you
>>>>>>> have any idea what it could be?
>>>>>>> (My connection timeout
>>>>>>> setting seems to have no
>>>>>>> effect on this 10s threshold)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/9/5 Ryan Lubke
>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>
>>>>>>> That is one solution.
>>>>>>> I'm working out an
>>>>>>> alternative right now.
>>>>>>> Stay tuned!
>>>>>>>
>>>>>>>>
>>>>>>>> Anyway it's not a
>>>>>>>> problem, I think the
>>>>>>>> FeedableBodyGenerator.feed()
>>>>>>>> method just has to
>>>>>>>> block until a context
>>>>>>>> has been (and
>>>>>>>> ThreadCache
>>>>>>>> initialized) to avoid
>>>>>>>> OOM errors
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien
>>>>>>>> Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What is very
>>>>>>>> strange is that I
>>>>>>>> tested with/without
>>>>>>>> the same
>>>>>>>> sessionCode with
>>>>>>>> our previous code,
>>>>>>>> the one not using
>>>>>>>> FeedableBodyGenerator,
>>>>>>>> which has a high
>>>>>>>> memory consumption.
>>>>>>>> Despites the fact
>>>>>>>> it had high memory
>>>>>>>> consumption, it
>>>>>>>> seems work fine to
>>>>>>>> upload multiple
>>>>>>>> documents if
>>>>>>>> allocated with a
>>>>>>>> large heap, and the
>>>>>>>> sessionCode seems
>>>>>>>> to have no effect.
>>>>>>>>
>>>>>>>> On the new impl
>>>>>>>> using the
>>>>>>>> FeedableBodyGenerator,
>>>>>>>> the sessionCode
>>>>>>>> sent as a multipart
>>>>>>>> bodypart seems to
>>>>>>>> have an effect.
>>>>>>>>
>>>>>>>> I have tried to
>>>>>>>> feed the queue
>>>>>>>> before sending the
>>>>>>>> request to AHC, but
>>>>>>>> this leads to this
>>>>>>>> exception
>>>>>>>> (with/without
>>>>>>>> sessionCode switching)
>>>>>>>> Caused by:
>>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>>> Timeout exceeded
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>>>>> at
>>>>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5 Sébastien
>>>>>>>> Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>> By the way, by
>>>>>>>> using a low
>>>>>>>> timeout with
>>>>>>>> the same
>>>>>>>> sessioncode, I
>>>>>>>> got the
>>>>>>>> following NPE:
>>>>>>>>
>>>>>>>> Caused by:
>>>>>>>> java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>> at
>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>
>>>>>>>>
>>>>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>>>>> httpCtx =
>>>>>>>>
>>>>>>>>
>>>>>>>> getHttpTransactionContext(c);
>>>>>>>>
>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>
>>>>>>>> I guess the
>>>>>>>> httpCtx is not
>>>>>>>> already
>>>>>>>> available to be
>>>>>>>> aborted
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/5
>>>>>>>> Sébastien
>>>>>>>> Lorber
>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is
>>>>>>>> right,
>>>>>>>> here's a
>>>>>>>> log I have
>>>>>>>> when I use
>>>>>>>> the same
>>>>>>>> session
>>>>>>>> code, ie
>>>>>>>> the remote
>>>>>>>> host is
>>>>>>>> blocking
>>>>>>>> the data or
>>>>>>>> something.
>>>>>>>> This is
>>>>>>>> obtained by
>>>>>>>> running 5
>>>>>>>> parallel
>>>>>>>> uploads.
>>>>>>>>
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 97
>>>>>>>> with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 100
>>>>>>>> with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 160
>>>>>>>> with
>>>>>>>> allowBlocking
>>>>>>>> = true*
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>> GC overhead
>>>>>>>> limit exceeded
>>>>>>>> Dumping
>>>>>>>> heap to
>>>>>>>> /ome/lorber/ureau/om
>>>>>>>> ...
>>>>>>>> Unable to
>>>>>>>> create
>>>>>>>> /ome/lorber/ureau/om:
>>>>>>>> Le fichier
>>>>>>>> existe
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Disconnected from
>>>>>>>> the target
>>>>>>>> VM,
>>>>>>>> address:
>>>>>>>> '127.0.0.1:49268
>>>>>>>> <http://127.0.0.1:49268>',
>>>>>>>> transport:
>>>>>>>> 'socket'
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Otherwise,
>>>>>>>> with
>>>>>>>> different
>>>>>>>> session
>>>>>>>> codes, I
>>>>>>>> get the
>>>>>>>> following:
>>>>>>>>
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> *Flushing
>>>>>>>> queue of
>>>>>>>> size 0 with
>>>>>>>> allowBlocking
>>>>>>>> = false*
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> Flushing
>>>>>>>> queue of
>>>>>>>> size 1 with
>>>>>>>> allowBlocking
>>>>>>>> = true
>>>>>>>> ... and
>>>>>>>> this
>>>>>>>> continues
>>>>>>>> without OOM
>>>>>>>>
>>>>>>>>
>>>>>>>> So, this
>>>>>>>> seems to be
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I think it
>>>>>>>> would be
>>>>>>>> great to be
>>>>>>>> able to be
>>>>>>>> able to
>>>>>>>> choose the
>>>>>>>> queue impl
>>>>>>>> behind
>>>>>>>> that FeedableBodyGenerator,
>>>>>>>> like I
>>>>>>>> suggested
>>>>>>>> in my pull
>>>>>>>> request.
>>>>>>>>
>>>>>>>> See here:
>>>>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>>>>
>>>>>>>> Using a
>>>>>>>> LinkedBlockingQueue
>>>>>>>> seems to be
>>>>>>>> a nice idea
>>>>>>>> in this
>>>>>>>> context,
>>>>>>>> and in my
>>>>>>>> case I
>>>>>>>> would
>>>>>>>> probably
>>>>>>>> use a queue
>>>>>>>> of size 1
>>>>>>>>
>>>>>>>>
>>>>>>>> This would
>>>>>>>> handle the
>>>>>>>> blocking of
>>>>>>>> the feed
>>>>>>>> method,
>>>>>>>> without
>>>>>>>> having to
>>>>>>>> use this:
>>>>>>>> if
>>>>>>>> (context !=
>>>>>>>> null) {
>>>>>>>>
>>>>>>>> flushQueue(true);
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Or perhaps
>>>>>>>> the feed()
>>>>>>>> method have
>>>>>>>> to wait
>>>>>>>> until a
>>>>>>>> context is
>>>>>>>> set in the
>>>>>>>> BodyGenerator ?
>>>>>>>>
>>>>>>>>
>>>>>>>> I think it
>>>>>>>> would be
>>>>>>>> more clear
>>>>>>>> if
>>>>>>>> the initializeAsynchronousTransfer
>>>>>>>> simply
>>>>>>>> didn't
>>>>>>>> flush the
>>>>>>>> queue but
>>>>>>>> just setup
>>>>>>>> the context.
>>>>>>>> Then the
>>>>>>>> feed method
>>>>>>>> would block
>>>>>>>> until
>>>>>>>> there's a
>>>>>>>> context
>>>>>>>> set, and
>>>>>>>> then flush
>>>>>>>> the queue
>>>>>>>> with
>>>>>>>> blocking
>>>>>>>> behavior.
>>>>>>>>
>>>>>>>> This is
>>>>>>>> probably
>>>>>>>> the next
>>>>>>>> step, but
>>>>>>>> as we are
>>>>>>>> using AHC
>>>>>>>> for async,
>>>>>>>> it would
>>>>>>>> probably be
>>>>>>>> great if
>>>>>>>> that
>>>>>>>> blocking
>>>>>>>> feed()
>>>>>>>> method was
>>>>>>>> called in a
>>>>>>>> worker
>>>>>>>> thread
>>>>>>>> instead of
>>>>>>>> our main
>>>>>>>> thread.
>>>>>>>> I won't use
>>>>>>>> this but
>>>>>>>> someone who
>>>>>>>> really
>>>>>>>> wants a
>>>>>>>> non-blocking impl
>>>>>>>> of
>>>>>>>> performant
>>>>>>>> multipart
>>>>>>>> fileupload
>>>>>>>> would
>>>>>>>> probably
>>>>>>>> need this,
>>>>>>>> or will use
>>>>>>>> an
>>>>>>>> ExecutorService
>>>>>>>> for the
>>>>>>>> feeding
>>>>>>>> operations
>>>>>>>> as a
>>>>>>>> workaround.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> again for
>>>>>>>> your reactivity
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/9/4
>>>>>>>> Ryan Lubke
>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sébastien
>>>>>>>> Lorber
>>>>>>>> wrote:
>>>>>>>>> I've
>>>>>>>>> integrated
>>>>>>>>> this
>>>>>>>>> change
>>>>>>>>> and it
>>>>>>>>> works
>>>>>>>>> fine
>>>>>>>>> except
>>>>>>>>> a
>>>>>>>>> little
>>>>>>>>> detail.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm
>>>>>>>>> uploading
>>>>>>>>> files
>>>>>>>>> to a
>>>>>>>>> third
>>>>>>>>> party
>>>>>>>>> API (a
>>>>>>>>> bit
>>>>>>>>> like S3).
>>>>>>>>> This
>>>>>>>>> API
>>>>>>>>> requires
>>>>>>>>> a
>>>>>>>>> "sessionCode"
>>>>>>>>> in
>>>>>>>>> each
>>>>>>>>> request.
>>>>>>>>> So
>>>>>>>>> there
>>>>>>>>> is a
>>>>>>>>> multipart
>>>>>>>>> StringPart
>>>>>>>>> with
>>>>>>>>> that
>>>>>>>>> SessionCode.
>>>>>>>>>
>>>>>>>>> We
>>>>>>>>> used
>>>>>>>>> to
>>>>>>>>> have a
>>>>>>>>> cache
>>>>>>>>> which
>>>>>>>>> holds
>>>>>>>>> the
>>>>>>>>> sessionCode
>>>>>>>>> 30min
>>>>>>>>> per
>>>>>>>>> user
>>>>>>>>> so
>>>>>>>>> that
>>>>>>>>> we do
>>>>>>>>> not
>>>>>>>>> need
>>>>>>>>> to
>>>>>>>>> init a
>>>>>>>>> new
>>>>>>>>> session each
>>>>>>>>> time.
>>>>>>>>>
>>>>>>>>> I had
>>>>>>>>> troubles
>>>>>>>>> in
>>>>>>>>> this
>>>>>>>>> very
>>>>>>>>> specific
>>>>>>>>> case:
>>>>>>>>> when I
>>>>>>>>> upload
>>>>>>>>> 5 docs
>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>> same
>>>>>>>>> session code.
>>>>>>>>> When I
>>>>>>>>> remove
>>>>>>>>> the
>>>>>>>>> cache
>>>>>>>>> and
>>>>>>>>> use 5
>>>>>>>>> different
>>>>>>>>> session codes,
>>>>>>>>> it
>>>>>>>>> works
>>>>>>>>> fine.
>>>>>>>>>
>>>>>>>>> I
>>>>>>>>> guess
>>>>>>>>> the
>>>>>>>>> remote
>>>>>>>>> service is
>>>>>>>>> blocking
>>>>>>>>> concurrent
>>>>>>>>> uploads with
>>>>>>>>> the
>>>>>>>>> same
>>>>>>>>> session code.
>>>>>>>>> I
>>>>>>>>> don't
>>>>>>>>> know
>>>>>>>>> at all.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Where
>>>>>>>>> I want
>>>>>>>>> to go
>>>>>>>>> is
>>>>>>>>> that I
>>>>>>>>> wouldn't
>>>>>>>>> have
>>>>>>>>> expected
>>>>>>>>> Grizzly to
>>>>>>>>> OOM
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Avertissement:
>>>>>>>>> Exception
>>>>>>>>> during
>>>>>>>>> FilterChain
>>>>>>>>> execution
>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>> Java
>>>>>>>>> heap space
>>>>>>>>> at
>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>> at
>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Caused
>>>>>>>>> by:
>>>>>>>>> java.util.concurrent.TimeoutException:
>>>>>>>>> null
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>> at
>>>>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>> at
>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> multipart.body.generator.feeder.buffer=100000
>>>>>>>>> ->
>>>>>>>>> size
>>>>>>>>> of
>>>>>>>>> each
>>>>>>>>> Buffer
>>>>>>>>> sent
>>>>>>>>> to the
>>>>>>>>> FeedableBodyGenerator
>>>>>>>>> transport.max.pending.bytes=1000000
>>>>>>>>> (I
>>>>>>>>> tried
>>>>>>>>> other
>>>>>>>>> settings,
>>>>>>>>> including
>>>>>>>>> AUTO_SIZE)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you
>>>>>>>>> have
>>>>>>>>> any
>>>>>>>>> idea
>>>>>>>>> why is
>>>>>>>>> there
>>>>>>>>> an OOM
>>>>>>>>> with
>>>>>>>>> these
>>>>>>>>> settings?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Perhaps it
>>>>>>>>> is
>>>>>>>>> because the
>>>>>>>>> feed()
>>>>>>>>> method
>>>>>>>>> of
>>>>>>>>> FeedableBodyGenerator
>>>>>>>>> doesn't block
>>>>>>>>> until
>>>>>>>>> the
>>>>>>>>> context is
>>>>>>>>> initialized.
>>>>>>>>>
>>>>>>>>> I
>>>>>>>>> guess
>>>>>>>>> the initializeAsynchronousTransfer
>>>>>>>>> is
>>>>>>>>> called
>>>>>>>>> only
>>>>>>>>> once
>>>>>>>>> the
>>>>>>>>> connection
>>>>>>>>> is
>>>>>>>>> established,
>>>>>>>>> and
>>>>>>>>> perhaps a
>>>>>>>>> lot of
>>>>>>>>> Buffer
>>>>>>>>> are
>>>>>>>>> added
>>>>>>>>> to the
>>>>>>>>> queue...
>>>>>>>> Yes,
>>>>>>>> it's
>>>>>>>> invoked
>>>>>>>> once
>>>>>>>> the
>>>>>>>> request
>>>>>>>> has
>>>>>>>> been
>>>>>>>> dispatched,
>>>>>>>> so if
>>>>>>>> the
>>>>>>>> generator
>>>>>>>> is fed
>>>>>>>> a lot
>>>>>>>> before
>>>>>>>> the
>>>>>>>> request, I
>>>>>>>> could
>>>>>>>> see
>>>>>>>> this
>>>>>>>> happening.
>>>>>>>> I'll
>>>>>>>> see
>>>>>>>> what I
>>>>>>>> can do
>>>>>>>> to
>>>>>>>> alleviate
>>>>>>>> that case.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> But
>>>>>>>>> I'm
>>>>>>>>> not
>>>>>>>>> sure
>>>>>>>>> at all
>>>>>>>>> because the
>>>>>>>>> session code
>>>>>>>>> is
>>>>>>>>> transmitted
>>>>>>>>> as a
>>>>>>>>> BodyPart
>>>>>>>>> and I
>>>>>>>>> get
>>>>>>>>> the
>>>>>>>>> same
>>>>>>>>> problem if
>>>>>>>>> i put
>>>>>>>>> it as
>>>>>>>>> the
>>>>>>>>> first
>>>>>>>>> or
>>>>>>>>> last
>>>>>>>>> multipart.
>>>>>>>>>
>>>>>>>>> It's
>>>>>>>>> not a
>>>>>>>>> big
>>>>>>>>> deal,
>>>>>>>>> perhaps I
>>>>>>>>> should
>>>>>>>>> always
>>>>>>>>> use a
>>>>>>>>> different
>>>>>>>>> session code
>>>>>>>>> for
>>>>>>>>> concurrent
>>>>>>>>> operations
>>>>>>>>> but
>>>>>>>>> I'd
>>>>>>>>> like
>>>>>>>>> to be
>>>>>>>>> sure
>>>>>>>>> that
>>>>>>>>> we
>>>>>>>>> won't
>>>>>>>>> have
>>>>>>>>> this
>>>>>>>>> issue
>>>>>>>>> in
>>>>>>>>> production...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/3
>>>>>>>>> Ryan
>>>>>>>>> Lubke
>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>
>>>>>>>>> Good
>>>>>>>>> catch.
>>>>>>>>> Fixed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sébastien
>>>>>>>>> Lorber
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's
>>>>>>>>>> a
>>>>>>>>>> little
>>>>>>>>>> mistake
>>>>>>>>>> in the
>>>>>>>>>> grizzly
>>>>>>>>>> ahc
>>>>>>>>>> provider
>>>>>>>>>> relative
>>>>>>>>>> to the
>>>>>>>>>> write
>>>>>>>>>> queue
>>>>>>>>>> size.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>>>>
>>>>>>>>>> As you
>>>>>>>>>> can
>>>>>>>>>> see,
>>>>>>>>>> the
>>>>>>>>>> TransportCustomizer
>>>>>>>>>> is called,
>>>>>>>>>> and
>>>>>>>>>> then
>>>>>>>>>> the
>>>>>>>>>> write
>>>>>>>>>> queue
>>>>>>>>>> size
>>>>>>>>>> (among
>>>>>>>>>> other
>>>>>>>>>> things)
>>>>>>>>>> is set
>>>>>>>>>> to AUTO_SIZE
>>>>>>>>>> (instead
>>>>>>>>>> of previously
>>>>>>>>>> UNLIMITED)
>>>>>>>>>>
>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>> think
>>>>>>>>>> the
>>>>>>>>>> default
>>>>>>>>>> settings
>>>>>>>>>> like
>>>>>>>>>> this
>>>>>>>>>> AUTO_SIZE
>>>>>>>>>> attribute
>>>>>>>>>> should
>>>>>>>>>> be set
>>>>>>>>>> before
>>>>>>>>>> the
>>>>>>>>>> customization
>>>>>>>>>> of the
>>>>>>>>>> transport,
>>>>>>>>>> or they
>>>>>>>>>> would
>>>>>>>>>> override
>>>>>>>>>> the
>>>>>>>>>> value
>>>>>>>>>> we customized.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This
>>>>>>>>>> is actually
>>>>>>>>>> my case,
>>>>>>>>>> since
>>>>>>>>>> I
>>>>>>>>>> can't
>>>>>>>>>> reproduce
>>>>>>>>>> my "bug"
>>>>>>>>>> which
>>>>>>>>>> is "high
>>>>>>>>>> memory
>>>>>>>>>> consumption",
>>>>>>>>>> even
>>>>>>>>>> when
>>>>>>>>>> using
>>>>>>>>>> -1 /
>>>>>>>>>> UNLIMITED
>>>>>>>>>> in the
>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This
>>>>>>>>>> could
>>>>>>>>>> work
>>>>>>>>>> fine
>>>>>>>>>> for
>>>>>>>>>> me with
>>>>>>>>>> AUTO_SIZE,
>>>>>>>>>> but
>>>>>>>>>> I'd
>>>>>>>>>> rather
>>>>>>>>>> be able
>>>>>>>>>> to tune
>>>>>>>>>> this
>>>>>>>>>> parameter
>>>>>>>>>> during
>>>>>>>>>> load
>>>>>>>>>> tests
>>>>>>>>>> to see
>>>>>>>>>> the
>>>>>>>>>> effect.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/31
>>>>>>>>>> Sebastien
>>>>>>>>>> Lorber
>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> i will
>>>>>>>>>> ckeck
>>>>>>>>>> that
>>>>>>>>>> on
>>>>>>>>>> monday.
>>>>>>>>>> I can
>>>>>>>>>> now
>>>>>>>>>> upload
>>>>>>>>>> a 500m
>>>>>>>>>> file
>>>>>>>>>> with
>>>>>>>>>> 40m
>>>>>>>>>> heap
>>>>>>>>>> size
>>>>>>>>>> ;)
>>>>>>>>>>
>>>>>>>>>> Envoyé
>>>>>>>>>> de
>>>>>>>>>> mon
>>>>>>>>>> iPhone
>>>>>>>>>>
>>>>>>>>>> Le
>>>>>>>>>> 30
>>>>>>>>>> août
>>>>>>>>>> 2013
>>>>>>>>>> à 20:49,
>>>>>>>>>> Ryan
>>>>>>>>>> Lubke
>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>> a écrit :
>>>>>>>>>>
>>>>>>>>>>> I'm
>>>>>>>>>>> going
>>>>>>>>>>> to
>>>>>>>>>>> be
>>>>>>>>>>> updating
>>>>>>>>>>> the
>>>>>>>>>>> Grizzly
>>>>>>>>>>> provider
>>>>>>>>>>> such
>>>>>>>>>>> that
>>>>>>>>>>> AUTO_SIZE
>>>>>>>>>>> (not
>>>>>>>>>>> AUTO_TUNE)
>>>>>>>>>>> is
>>>>>>>>>>> the
>>>>>>>>>>> default,
>>>>>>>>>>> so
>>>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>> avoid
>>>>>>>>>>> the
>>>>>>>>>>> use
>>>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>> TransportCustomizer.
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Regarding
>>>>>>>>>>>> your
>>>>>>>>>>>> tuning
>>>>>>>>>>>> question,
>>>>>>>>>>>> I would
>>>>>>>>>>>> probably
>>>>>>>>>>>> set
>>>>>>>>>>>> the
>>>>>>>>>>>> value
>>>>>>>>>>>> to
>>>>>>>>>>>> AsyncQueueWriter.AUTO_TUNE
>>>>>>>>>>>> (this
>>>>>>>>>>>> will
>>>>>>>>>>>> be
>>>>>>>>>>>> four
>>>>>>>>>>>> times
>>>>>>>>>>>> the
>>>>>>>>>>>> socket
>>>>>>>>>>>> write
>>>>>>>>>>>> buffer)
>>>>>>>>>>>> and
>>>>>>>>>>>> see
>>>>>>>>>>>> how
>>>>>>>>>>>> that
>>>>>>>>>>>> works.
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>> Lubke
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> A question
>>>>>>>>>>>>> first.
>>>>>>>>>>>>> With
>>>>>>>>>>>>> these
>>>>>>>>>>>>> changes,
>>>>>>>>>>>>> your
>>>>>>>>>>>>> memory
>>>>>>>>>>>>> usage
>>>>>>>>>>>>> is
>>>>>>>>>>>>> more
>>>>>>>>>>>>> inline
>>>>>>>>>>>>> with
>>>>>>>>>>>>> what
>>>>>>>>>>>>> you
>>>>>>>>>>>>> were
>>>>>>>>>>>>> looking
>>>>>>>>>>>>> for?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> By
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> idea
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> 1.7.20
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> released
>>>>>>>>>>>>>> (with
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>> improvements?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> know
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> a release
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> install
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>> own
>>>>>>>>>>>>>> temp
>>>>>>>>>>>>>> release
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> Nexus
>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/30
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank
>>>>>>>>>>>>>> you,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> works
>>>>>>>>>>>>>> fine!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just
>>>>>>>>>>>>>> had
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> modify
>>>>>>>>>>>>>> a single
>>>>>>>>>>>>>> line
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> commit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>>>>> ->
>>>>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>> I let
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> (-1)
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>> block,
>>>>>>>>>>>>>> canWrite
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> returns
>>>>>>>>>>>>>> true
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Btw,
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>> find
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> pass
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> a config
>>>>>>>>>>>>>> attribute,
>>>>>>>>>>>>>> neither
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> size
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> write
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> talked
>>>>>>>>>>>>>> about.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> end,
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> current
>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> "canWrite
>>>>>>>>>>>>>> = false"
>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>> not,
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> please
>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> v1.7.20
>>>>>>>>>>>>>> ? thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>> does
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>> sens
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> number
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>> un
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> feed(Buffer)
>>>>>>>>>>>>>> method
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>>>> ? do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> tuning
>>>>>>>>>>>>>> recommandation?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please
>>>>>>>>>>>>>> disregard.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>> a sample
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>> performing
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> feed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> -rl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>>> recommend
>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> Connection.canWrite()
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener)
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>>> default,
>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> four
>>>>>>>>>>>>>>>> times
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> buffer).
>>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> exceeds
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> value,
>>>>>>>>>>>>>>>> canWrite()
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>> false.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> occurs,
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>>> a WriteListener
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> notified
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> length
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> configured
>>>>>>>>>>>>>>>> max
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> simulate
>>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>>> callback
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>> invoked.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> FutureImpl<Boolean>
>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>> = Futures.createSafeFuture();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> //
>>>>>>>>>>>>>>>> Connection
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> obtained
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> calling
>>>>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> connection.notifyCanWrite(new
>>>>>>>>>>>>>>>> WriteHandler()
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>>>>> throws
>>>>>>>>>>>>>>>> Exception
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> void
>>>>>>>>>>>>>>>> onError(Throwable
>>>>>>>>>>>>>>>> t)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> });
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>>> writeTimeout
>>>>>>>>>>>>>>>> = 30;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>>> (ExecutionException
>>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch
>>>>>>>>>>>>>>>> (Exception
>>>>>>>>>>>>>>>> e)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> did
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> a blocking
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> totally
>>>>>>>>>>>>>>>>> useless
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>> consuming
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>> blocks
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> feeding,
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> intention
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> beginning.
>>>>>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>>> strategy
>>>>>>>>>>>>>>>>> used?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I use
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> use SameThreadIOStrategy
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>> a 70m
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 50m,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> put
>>>>>>>>>>>>>>>>> a Thread.sleep(30)
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> I know.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>>> things
>>>>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send.
>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> possible?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> 100mo
>>>>>>>>>>>>>>>>> heap,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>> a sleep
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 100ms
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By
>>>>>>>>>>>>>>>>> chance
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> MessageCloner
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>> filter?
>>>>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> cloned,
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> I remove
>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>> How?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> trying
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> a 500m
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> cases
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> files
>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> consumption
>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> x parallel
>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> according
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> activity.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>> reduced
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> footprint
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>> allocation
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> concerned,
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> untunable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That
>>>>>>>>>>>>>>>>> said,
>>>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>>> free
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>> a feature
>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>>> size.
>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>> application
>>>>>>>>>>>>>>>>> specific.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> curious,
>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> sending?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> buffers
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a size
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> 33842
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> limit
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> near
>>>>>>>>>>>>>>>>>> half
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> capacity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> tune
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> consumes
>>>>>>>>>>>>>>>>>> less
>>>>>>>>>>>>>>>>>> memory?
>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> ideal
>>>>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> reviewing
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>>> today,
>>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>> again!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> OOM:
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> stands
>>>>>>>>>>>>>>>>>> now,
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> SSLFilter,
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>> a buffer
>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> order
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> accommodate
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> encrypted
>>>>>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Depending
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>> endpoints,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> becoming
>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> situation,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>> pretty
>>>>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>>>>> shielded
>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>> I will
>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> leverage
>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> With
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>> usecase
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> works,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> uploaded
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>>> a big
>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> body
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> that?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>>> memory,
>>>>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>>>>> of:
>>>>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> SSLUtils$3
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> BufferBuffers
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> WriteResult
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> exemple
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>>> stacktrace:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>>>>>> idea?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Excellent!
>>>>>>>>>>>>>>>>>>> Looking
>>>>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>>>>> request!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> works
>>>>>>>>>>>>>>>>>>>> fine,
>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>> a pull
>>>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>> tomorrow
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> a better
>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>> Part
>>>>>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I created
>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> OutputStream
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> redirects
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>>>>> feeder.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>> I currently
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> feeder
>>>>>>>>>>>>>>>>>>>> feeds
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>>>> polling
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>> I need
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> expose
>>>>>>>>>>>>>>>>>>>> a limit
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> something,
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> a thread
>>>>>>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> filepart
>>>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes,
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>> I was
>>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> tackle
>>>>>>>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>>>>>> up
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok
>>>>>>>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think
>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>>> do,
>>>>>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>>>>> bodyGenerator
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Request
>>>>>>>>>>>>>>>>>>>>> uploadRequest1
>>>>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ListenableFuture<Response>
>>>>>>>>>>>>>>>>>>>>> asyncRes
>>>>>>>>>>>>>>>>>>>>> = asyncHttpClient
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Response
>>>>>>>>>>>>>>>>>>>>> uploadResponse
>>>>>>>>>>>>>>>>>>>>> = asyncRes.get();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I guess
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> interesting
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/8/26
>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> / Grizzly
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> streaming,
>>>>>>>>>>>>>>>>>>>>> I mean
>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>>>>> a byte[]
>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> contans
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file,
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>>>>> a large
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>>>>> tryied
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> a Heap
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> ByteBuffer
>>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>> managers,
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> reallocate=true/false
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> success.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> content
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> appended
>>>>>>>>>>>>>>>>>>>>> wto
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> BufferOutputStream,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> underlying
>>>>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>>> uploading
>>>>>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> Grizzly?
>>>>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>>>>>>>> tied
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> Multipart
>>>>>>>>>>>>>>>>>>>>> uploads,
>>>>>>>>>>>>>>>>>>>>> you'd
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> convert
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> multipart
>>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> Buffers
>>>>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>> a closer
>>>>>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> area
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Btw
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> upload.
>>>>>>>>>>>>>>>>>>>>> I receive
>>>>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> transmit
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> a storage
>>>>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>>>>> (like
>>>>>>>>>>>>>>>>>>>>> S3).
>>>>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>> bevause
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> streaming
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>>>>>> fle
>>>>>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> system,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> provides
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> input
>>>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>