Okay, I've committed another set of refactorings to the
FeedableBodyGenerator.
For your use case, you should extend FeedableBodyGenerator.SimpleFeeder.
Let me know if you run into issues.
Sébastien Lorber wrote:
> Yes I think it would, so that I can feed the queue at once
>
> One thread will be locked during the feeding for nothing but it's not
> a real problem in my usecase.
>
>
> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
> <mailto:ryan.lubke_at_oracle.com>>
>
> Would having a different listener that will be notified once async
> transferring has been started work better for you?
>
> Something like:
>
> onAsyncTransferInitiated() {
> // invoke your feed method
> }
>
> ?
>
>
>
> Sébastien Lorber wrote:
>> Unfortunatly I won't be able to use the Feeder non-blocking stuff
>> for now, because of how the multipart request in handled in AHC
>>
>>
>> Here's my feeding method:
>>
>> public void feed() throws IOException {
>> Part[] partsArray = parts.toArray(new Part[parts.size()]);
>> try ( OutputStream outputStream = createFeedingOutputStream() ) {
>> Part.sendParts(outputStream,partsArray,multipartBoundary);
>> } catch (Exception e) {
>> throw new IllegalStateException("Unable to feed the
>> FeedableBodyGenerator",e);
>> }
>> }
>>
>>
>> As you can see, the multipart Parts array can only be pushed to
>> the OutputStream, I don't have any way to "pull" the data when
>> the canFeed() method is triggered.
>>
>>
>> But I've seen that there's
>> a com.ning.http.multipart.MultipartBody#read that seems to
>> provide a memory efficient way to pull data from a Multipart body...
>>
>> Should see what I come up with this
>>
>>
>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
>> <mailto:lorber.sebastien_at_gmail.com>>
>>
>> It seems the Feeder is highly recommended but not mandatory
>> so I tried without.
>>
>> With my existing code it seems there is a synchronization
>> problem.
>>
>>
>> The feeding threads get locked to the prematureFeed.get();
>>
>> So the Grizzly kernel threads are unable to acquire the lock
>> required to enter the initializeAsynchronousTransfer method
>>
>>
>>
>> Will try with an implementation of Feeder
>>
>>
>>
>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
>> <mailto:lorber.sebastien_at_gmail.com>>
>>
>> Hmmm it seems I have a problem with one of your maven
>> plugins. I'll try to bypass it, but for info:
>>
>> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
>> [WARNING]
>> [WARNING] Some problems were encountered while building
>> the effective settings
>> [WARNING]
>> 'profiles.profile[default].repositories.repository.id
>> <http://repositories.repository.id>' must be unique but
>> found duplicate repository with id
>> fullsix-maven-repository @ /home/slorber/.m2/settings.xml
>> [WARNING]
>> [INFO] Scanning for projects...
>> [INFO]
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO]
>> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean)
>> @ async-http-client ---
>> [INFO]
>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>> (enforce-maven) @ async-http-client ---
>> [INFO]
>> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
>> (enforce-versions) @ async-http-client ---
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.4.3:resources
>> (default-resources) @ async-http-client ---
>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> [INFO] skip non existing resourceDirectory
>> /home/slorber/Bureau/ahc2/src/main/resources
>> [INFO]
>> [INFO] --- maven-compiler-plugin:2.3.2:compile
>> (default-compile) @ async-http-client ---
>> [INFO] Compiling 158 source files to
>> /home/slorber/Bureau/ahc2/target/classes
>> [INFO]
>> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
>> (check-java-1.5-compat) @ async-http-client ---*
>> *[INFO] Checking unresolved references to
>> org.codehaus.mojo.signature:java15:1.0*
>> *[ERROR] Undefined reference:
>> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
>> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class*
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 8.747s
>> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
>> [INFO] Final Memory: 30M/453M
>> [INFO]
>> ------------------------------------------------------------------------
>> *[ERROR] Failed to execute goal
>> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
>> (check-java-1.5-compat) on project async-http-client:
>> Signature errors found. Verify them and put
>> @IgnoreJRERequirement on them. -> [Help 1]*
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run
>> Maven with the -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full
>> debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and
>> possible solutions, please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>
>>
>>
>> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
>> <mailto:lorber.sebastien_at_gmail.com>>
>>
>> Ok thank you, I'll try to implement that today and
>> will give you my feedback :)
>>
>>
>> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
>> <mailto:ryan.lubke_at_oracle.com>>
>>
>> Okay,
>>
>> I've committed my initial changes to the AHC
>> repository. Here's a summary of the changes:
>>
>>
>> *Improvements to the FeedableBodyGenerator
>> (Grizzly's).
>> - Don't allow queueing of data before
>> initiateAsyncTransfer has been invoked. In low memory
>> heaps, this could lead to an OOM if the source is
>> feeding too fast. The new behavior is to
>> block until initiateAsyncTransfer is called, at
>> which time the blocked thread may proceed with
>> the feed operation.
>> - Introduce the concept of a Feeder.
>> Implementations are responsible, at a high level,
>> for:
>> + letting the provider know that data is
>> available to be fed without blocking
>> + allowing the registration of a callback that
>> the Feeder implementation may invoke
>> to signal that more data is available, if it
>> wasn't available at a previous point in time.
>> - When using a Feeder with a secure request, the
>> SSL handshake will be kicked off by the
>> initiateAsyncTransfer call, but feeding of data
>> will not occur until the handshake is complete.
>> This is necessary as the SSLFilter will queue up
>> all writes until the handshake is complete,
>> and currently, the buffer isn't tied in with the
>> transport flow control mechanism.
>> NOTE: This new SSL behavior is not currently
>> applied when invoking the feed() method
>> outside the context of a Feeder. Still need to
>> address that.
>> - Exposed configuration of the async write queue
>> limit through the FeedableBodyGenerator.
>> This is an improvement on using a
>> TransportCustomizer as any configuration there is
>> transport-wide, and therefor applied to all
>> Connections. By exposing it here, each feeder
>> may have a different byte limit.
>> - Improved documentation for this class*
>>
>>
>>
>> I recommend reading through the javadoc comments
>> in the source [1] for FeedableBodyGenerator
>> (comments welcome).
>> Additionally, I would re-work your code to
>> leverage the Feeder instead of calling feed()
>> directly.
>>
>> If you have issues implementing Feeder, do let us
>> know.
>>
>> If you have additional questions, again, let us know.
>>
>> Thanks,
>> -rl
>>
>> [1]
>> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>
>>
>>
>> Ryan Lubke wrote:
>>>
>>>
>>> Sébastien Lorber wrote:
>>>> So in the end I've end up with an
>>>> implementation that's working for me.
>>>>
>>>>
>>>> I think there are 2 bugs:
>>>>
>>>> 1) The bytes can accumulate in the
>>>> FeedableBodyGenerator queue if the
>>>> initialize(ctx) method is not called fast enough.
>>>> This can be solved by using a BlockingQueue of
>>>> size 1 and the put() method.
>>>>
>>>> 2) Once the context is injected, the
>>>> FeedableBodyGenerator flushes the queue.
>>>> The matter is that if the connection is new,
>>>> not warmed up by a previous request, then the
>>>> SSL handshake is not done yet, and it seems
>>>> that the bytes are accumulated in some part of
>>>> the SSL filter which doesn't deliver them to
>>>> the connection until the handshake has
>>>> completed, so c.canWrite() continues to return
>>>> true.
>>>>
>>>>
>>>>
>>>>
>>>> I have replaced some part of the
>>>> FeedableBodyGenerator to test this and it works
>>>> pretty fine. See what I have changed:
>>>>
>>>> 1)
>>>> private final BlockingQueue<BodyPart> queue
>>>> = new LinkedBlockingQueue<BodyPart>(1);
>>>>
>>>>
>>>> 2)
>>>> public void feed(final Buffer buffer, final
>>>> boolean last) throws IOException {
>>>> try {
>>>> queue.put(new BodyPart(buffer, last));
>>>> } catch (InterruptedException e) {
>>>> throw new RuntimeException(e);
>>>> }
>>>> queueSize.incrementAndGet();
>>>> if (context != null) {
>>>>
>>>> blockUntilConnectionIsReadyToWrite(context);
>>>> flushQueue(true);
>>>> }
>>>> }
>>>>
>>>> private void
>>>> blockUntilConnectionIsReadyToWrite(FilterChainContext
>>>> fcc) {
>>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>>> try { Thread.sleep(10); } catch (
>>>> Exception e ) { throw new RuntimeException(e); }
>>>> }
>>>> }
>>>> }
>>>>
>>>> private boolean
>>>> connectionIsReadyToWrite(FilterChainContext fcc) {
>>>> Connection connection =
>>>> fcc.getConnection();
>>>> SSLEngine sslEngine =
>>>> SSLUtils.getSSLEngine(connection);
>>>> return sslEngine != null &&
>>>> !SSLUtils.isHandshaking(sslEngine);
>>>> }
>>>>
>>>>
>>>>
>>>> What do you think?
>>>
>>> We had come to similar conclusions on this end.
>>> I'm still working through testing the idea I
>>> mentioned previously (took longer than I
>>> expected - sorry).
>>> I hope to have something for you to test very soon.
>>>
>>> Note that it will be taking the above into
>>> account as well.
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/5 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>>
>>>> I have tried to put a while ( context ==
>>>> null ) Thread.sleep but it doesn't seem to
>>>> work, when the context gets injected, after
>>>> the sleeps, there's an OOM
>>>>
>>>> So I hope you'll have more success with
>>>> your alternative :)
>>>>
>>>>
>>>>
>>>> I have done another test, remember my code
>>>> that worked, which previously "warmed" the
>>>> Thread with an useless request.
>>>>
>>>> private Runnable
>>>> uploadSingleDocumentRunnable = new Runnable() {
>>>> @Override
>>>> public void run() {
>>>> try {
>>>> getUselessSessionCode();
>>>> Thread.sleep(X);
>>>> uploadSingleDocument();
>>>> } catch ( Exception e ) {
>>>> throw new RuntimeException("file
>>>> upload failed",e);
>>>> }
>>>> }
>>>> };
>>>>
>>>> I have put a sleep of X between the useless
>>>> warmup request, and the real upload request
>>>>
>>>>
>>>> What I noticed is that there is a very
>>>> different behavior according to the value of X
>>>>
>>>>
>>>> Under 10 seconds, it seems the stuff is
>>>> still warm, I can upload the documents.
>>>> Around 10 seconds I get a stack which seems
>>>> to be "connection closed" or something
>>>> Above 10 seconds, I get OOM like if the
>>>> stuff wasn't warm.
>>>>
>>>>
>>>> The stacks I get for 10 seconds looks like
>>>>
>>>> Caused by: javax.net.ssl.SSLException:
>>>> SSLEngine is CLOSED
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>> at
>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>>
>>>> I think I got some other different stacks
>>>> saying Connection Closed Remotely or
>>>> something like that.
>>>>
>>>>
>>>>
>>>>
>>>> So it seems that something is bound to my
>>>> thread, and it stays bound to it for about
>>>> 10 seconds, do you have any idea what it
>>>> could be?
>>>> (My connection timeout setting seems to
>>>> have no effect on this 10s threshold)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com
>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>
>>>> That is one solution. I'm working out
>>>> an alternative right now. Stay tuned!
>>>>
>>>>>
>>>>> Anyway it's not a problem, I think the
>>>>> FeedableBodyGenerator.feed() method
>>>>> just has to block until a context has
>>>>> been (and ThreadCache initialized) to
>>>>> avoid OOM errors
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/5 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>>
>>>>> What is very strange is that I
>>>>> tested with/without the same
>>>>> sessionCode with our previous
>>>>> code, the one not using
>>>>> FeedableBodyGenerator, which has a
>>>>> high memory consumption.
>>>>> Despites the fact it had high
>>>>> memory consumption, it seems work
>>>>> fine to upload multiple documents
>>>>> if allocated with a large heap,
>>>>> and the sessionCode seems to have
>>>>> no effect.
>>>>>
>>>>> On the new impl using the
>>>>> FeedableBodyGenerator, the
>>>>> sessionCode sent as a multipart
>>>>> bodypart seems to have an effect.
>>>>>
>>>>> I have tried to feed the queue
>>>>> before sending the request to AHC,
>>>>> but this leads to this exception
>>>>> (with/without sessionCode switching)
>>>>> Caused by:
>>>>> java.util.concurrent.TimeoutException:
>>>>> Timeout exceeded
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>>> at
>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>>> at
>>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>>> at
>>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/5 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>> By the way, by using a low
>>>>> timeout with the same
>>>>> sessioncode, I got the
>>>>> following NPE:
>>>>>
>>>>> Caused by:
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>
>>>>>
>>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>>> httpCtx =
>>>>>
>>>>> getHttpTransactionContext(c);
>>>>>
>>>>> httpCtx.abort(e.getCause());
>>>>>
>>>>> I guess the httpCtx is not
>>>>> already available to be aborted
>>>>>
>>>>>
>>>>> 2013/9/5 Sébastien Lorber
>>>>> <lorber.sebastien_at_gmail.com
>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>
>>>>>
>>>>>
>>>>> This is right, here's a
>>>>> log I have when I use the
>>>>> same session code, ie the
>>>>> remote host is blocking
>>>>> the data or something.
>>>>> This is obtained by
>>>>> running 5 parallel uploads.
>>>>>
>>>>> *Flushing queue of size 0
>>>>> with allowBlocking = false*
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> *Flushing queue of size 97
>>>>> with allowBlocking = false*
>>>>> *Flushing queue of size
>>>>> 100 with allowBlocking =
>>>>> false*
>>>>> *Flushing queue of size
>>>>> 160 with allowBlocking = true*
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 0
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> java.lang.OutOfMemoryError: GC
>>>>> overhead limit exceeded
>>>>> Dumping heap to
>>>>> /ome/lorber/ureau/om ...
>>>>> Unable to create
>>>>> /ome/lorber/ureau/om: Le
>>>>> fichier existe
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Disconnected from the
>>>>> target VM, address:
>>>>> '127.0.0.1:49268
>>>>> <http://127.0.0.1:49268>',
>>>>> transport: 'socket'
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Otherwise, with different
>>>>> session codes, I get the
>>>>> following:
>>>>>
>>>>> *Flushing queue of size 0
>>>>> with allowBlocking = false*
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> *Flushing queue of size 0
>>>>> with allowBlocking = false*
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> *Flushing queue of size 0
>>>>> with allowBlocking = false*
>>>>> *Flushing queue of size 0
>>>>> with allowBlocking = false*
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> Flushing queue of size 1
>>>>> with allowBlocking = true
>>>>> ... and this continues
>>>>> without OOM
>>>>>
>>>>>
>>>>> So, this seems to be the
>>>>> problem.
>>>>>
>>>>>
>>>>>
>>>>> I think it would be great
>>>>> to be able to be able to
>>>>> choose the queue impl
>>>>> behind
>>>>> that FeedableBodyGenerator, like
>>>>> I suggested in my pull
>>>>> request.
>>>>>
>>>>> See here:
>>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>>
>>>>> Using a
>>>>> LinkedBlockingQueue seems
>>>>> to be a nice idea in this
>>>>> context, and in my case I
>>>>> would probably use a queue
>>>>> of size 1
>>>>>
>>>>>
>>>>> This would handle the
>>>>> blocking of the feed
>>>>> method, without having to
>>>>> use this:
>>>>> if (context != null) {
>>>>> flushQueue(true);
>>>>> }
>>>>>
>>>>>
>>>>> Or perhaps the feed()
>>>>> method have to wait until
>>>>> a context is set in the
>>>>> BodyGenerator ?
>>>>>
>>>>>
>>>>> I think it would be more
>>>>> clear if
>>>>> the initializeAsynchronousTransfer
>>>>> simply didn't flush the
>>>>> queue but just setup the
>>>>> context.
>>>>> Then the feed method would
>>>>> block until there's a
>>>>> context set, and then
>>>>> flush the queue with
>>>>> blocking behavior.
>>>>>
>>>>> This is probably the next
>>>>> step, but as we are using
>>>>> AHC for async, it would
>>>>> probably be great if that
>>>>> blocking feed() method was
>>>>> called in a worker thread
>>>>> instead of our main thread.
>>>>> I won't use this but
>>>>> someone who really wants a
>>>>> non-blocking impl of
>>>>> performant multipart
>>>>> fileupload would probably
>>>>> need this, or will use an
>>>>> ExecutorService for the
>>>>> feeding operations as a
>>>>> workaround.
>>>>>
>>>>>
>>>>>
>>>>> Thanks again for your
>>>>> reactivity
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/4 Ryan Lubke
>>>>> <ryan.lubke_at_oracle.com
>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>> I've integrated this
>>>>>> change and it works
>>>>>> fine except a little
>>>>>> detail.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I'm uploading files
>>>>>> to a third party API
>>>>>> (a bit like S3).
>>>>>> This API requires a
>>>>>> "sessionCode" in each
>>>>>> request. So there is
>>>>>> a multipart
>>>>>> StringPart with that
>>>>>> SessionCode.
>>>>>>
>>>>>> We used to have a
>>>>>> cache which holds the
>>>>>> sessionCode 30min per
>>>>>> user so that we do
>>>>>> not need to init a
>>>>>> new session each time.
>>>>>>
>>>>>> I had troubles in
>>>>>> this very specific
>>>>>> case: when I upload 5
>>>>>> docs with the same
>>>>>> session code.
>>>>>> When I remove the
>>>>>> cache and use 5
>>>>>> different session
>>>>>> codes, it works fine.
>>>>>>
>>>>>> I guess the remote
>>>>>> service is blocking
>>>>>> concurrent uploads
>>>>>> with the same session
>>>>>> code. I don't know at
>>>>>> all.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Where I want to go is
>>>>>> that I wouldn't have
>>>>>> expected Grizzly to OOM
>>>>>>
>>>>>>
>>>>>> Avertissement:
>>>>>> Exception during
>>>>>> FilterChain execution
>>>>>> java.lang.OutOfMemoryError:
>>>>>> Java heap space
>>>>>> at
>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>> at
>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>>> at
>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>> at
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>> at
>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>> at
>>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>>> at
>>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Caused by:
>>>>>> java.util.concurrent.TimeoutException:
>>>>>> null
>>>>>> at
>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>> at
>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>>
>>>>>>
>>>>>>
>>>>>> multipart.body.generator.feeder.buffer=100000
>>>>>> -> size of each
>>>>>> Buffer sent to the
>>>>>> FeedableBodyGenerator
>>>>>> transport.max.pending.bytes=1000000
>>>>>> (I tried other
>>>>>> settings, including
>>>>>> AUTO_SIZE)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Do you have any idea
>>>>>> why is there an OOM
>>>>>> with these settings?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Perhaps it is because
>>>>>> the feed() method of
>>>>>> FeedableBodyGenerator
>>>>>> doesn't block until
>>>>>> the context is
>>>>>> initialized.
>>>>>> I guess
>>>>>> the initializeAsynchronousTransfer
>>>>>> is called only once
>>>>>> the connection is
>>>>>> established, and
>>>>>> perhaps a lot of
>>>>>> Buffer are added to
>>>>>> the queue...
>>>>> Yes, it's invoked once
>>>>> the request has been
>>>>> dispatched, so if the
>>>>> generator is fed a lot
>>>>> before the request, I
>>>>> could see this happening.
>>>>> I'll see what I can do
>>>>> to alleviate that case.
>>>>>
>>>>>>
>>>>>> But I'm not sure at
>>>>>> all because the
>>>>>> session code is
>>>>>> transmitted as a
>>>>>> BodyPart and I get
>>>>>> the same problem if i
>>>>>> put it as the first
>>>>>> or last multipart.
>>>>>>
>>>>>> It's not a big deal,
>>>>>> perhaps I should
>>>>>> always use a
>>>>>> different session
>>>>>> code for concurrent
>>>>>> operations but I'd
>>>>>> like to be sure that
>>>>>> we won't have this
>>>>>> issue in production...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/9/3 Ryan Lubke
>>>>>> <ryan.lubke_at_oracle.com <mailto:ryan.lubke_at_oracle.com>>
>>>>>>
>>>>>> Good catch. Fixed.
>>>>>>
>>>>>>
>>>>>> Sébastien Lorber
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> There's a little
>>>>>>> mistake in the
>>>>>>> grizzly ahc
>>>>>>> provider
>>>>>>> relative to the
>>>>>>> write queue size.
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>>
>>>>>>> As you can see,
>>>>>>> the
>>>>>>> TransportCustomizer
>>>>>>> is called, and
>>>>>>> then the write
>>>>>>> queue size
>>>>>>> (among other
>>>>>>> things) is set
>>>>>>> to AUTO_SIZE
>>>>>>> (instead of
>>>>>>> previously
>>>>>>> UNLIMITED)
>>>>>>>
>>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think the
>>>>>>> default settings
>>>>>>> like this
>>>>>>> AUTO_SIZE
>>>>>>> attribute should
>>>>>>> be set before
>>>>>>> the
>>>>>>> customization of
>>>>>>> the transport,
>>>>>>> or they would
>>>>>>> override the
>>>>>>> value we customized.
>>>>>>>
>>>>>>>
>>>>>>> This is actually
>>>>>>> my case, since I
>>>>>>> can't reproduce
>>>>>>> my "bug" which
>>>>>>> is "high memory
>>>>>>> consumption",
>>>>>>> even when using
>>>>>>> -1 / UNLIMITED
>>>>>>> in the
>>>>>>> TransportCustomizer.
>>>>>>>
>>>>>>>
>>>>>>> This could work
>>>>>>> fine for me with
>>>>>>> AUTO_SIZE, but
>>>>>>> I'd rather be
>>>>>>> able to tune
>>>>>>> this parameter
>>>>>>> during load
>>>>>>> tests to see the
>>>>>>> effect.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/31
>>>>>>> Sebastien Lorber
>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>
>>>>>>> Thanks i
>>>>>>> will ckeck
>>>>>>> that on
>>>>>>> monday. I
>>>>>>> can now
>>>>>>> upload a
>>>>>>> 500m file
>>>>>>> with 40m
>>>>>>> heap size ;)
>>>>>>>
>>>>>>> Envoyé de
>>>>>>> mon iPhone
>>>>>>>
>>>>>>> Le 30 août
>>>>>>> 2013 à
>>>>>>> 20:49, Ryan
>>>>>>> Lubke
>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>> a écrit :
>>>>>>>
>>>>>>>> I'm going
>>>>>>>> to be
>>>>>>>> updating
>>>>>>>> the Grizzly
>>>>>>>> provider
>>>>>>>> such that
>>>>>>>> AUTO_SIZE
>>>>>>>> (not
>>>>>>>> AUTO_TUNE)
>>>>>>>> is the
>>>>>>>> default, so
>>>>>>>> you can
>>>>>>>> avoid the
>>>>>>>> use of the
>>>>>>>> TransportCustomizer.
>>>>>>>>
>>>>>>>> Ryan Lubke
>>>>>>>> wrote:
>>>>>>>>> Regarding
>>>>>>>>> your
>>>>>>>>> tuning
>>>>>>>>> question,
>>>>>>>>> I would
>>>>>>>>> probably
>>>>>>>>> set the
>>>>>>>>> value to
>>>>>>>>> AsyncQueueWriter.AUTO_TUNE
>>>>>>>>> (this will
>>>>>>>>> be four
>>>>>>>>> times the
>>>>>>>>> socket
>>>>>>>>> write
>>>>>>>>> buffer)
>>>>>>>>> and see
>>>>>>>>> how that
>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> Ryan Lubke
>>>>>>>>> wrote:
>>>>>>>>>> A
>>>>>>>>>> question
>>>>>>>>>> first.
>>>>>>>>>> With
>>>>>>>>>> these
>>>>>>>>>> changes,
>>>>>>>>>> your
>>>>>>>>>> memory
>>>>>>>>>> usage is
>>>>>>>>>> more
>>>>>>>>>> inline
>>>>>>>>>> with what
>>>>>>>>>> you were
>>>>>>>>>> looking for?
>>>>>>>>>>
>>>>>>>>>> Sébastien
>>>>>>>>>> Lorber
>>>>>>>>>> wrote:
>>>>>>>>>>> By the
>>>>>>>>>>> way, do
>>>>>>>>>>> you have
>>>>>>>>>>> any idea
>>>>>>>>>>> when the
>>>>>>>>>>> 1.7.20
>>>>>>>>>>> will be
>>>>>>>>>>> released
>>>>>>>>>>> (with
>>>>>>>>>>> these
>>>>>>>>>>> new
>>>>>>>>>>> improvements?)
>>>>>>>>>>>
>>>>>>>>>>> We would
>>>>>>>>>>> like to
>>>>>>>>>>> know if
>>>>>>>>>>> we wait
>>>>>>>>>>> for a
>>>>>>>>>>> release
>>>>>>>>>>> or if we
>>>>>>>>>>> install
>>>>>>>>>>> our own
>>>>>>>>>>> temp
>>>>>>>>>>> release
>>>>>>>>>>> on Nexus :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/8/30 Sébastien
>>>>>>>>>>> Lorber
>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>> it
>>>>>>>>>>> works fine!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I
>>>>>>>>>>> just
>>>>>>>>>>> had
>>>>>>>>>>> to
>>>>>>>>>>> modify
>>>>>>>>>>> a
>>>>>>>>>>> single
>>>>>>>>>>> line
>>>>>>>>>>> after your
>>>>>>>>>>> commit.
>>>>>>>>>>>
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>>> ->
>>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If I
>>>>>>>>>>> let
>>>>>>>>>>> the
>>>>>>>>>>> initial
>>>>>>>>>>> value (-1)
>>>>>>>>>>> it
>>>>>>>>>>> won't block,
>>>>>>>>>>> canWrite
>>>>>>>>>>> always
>>>>>>>>>>> returns
>>>>>>>>>>> true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Btw,
>>>>>>>>>>> on
>>>>>>>>>>> AHC
>>>>>>>>>>> I
>>>>>>>>>>> didn't
>>>>>>>>>>> find
>>>>>>>>>>> any
>>>>>>>>>>> way
>>>>>>>>>>> to
>>>>>>>>>>> pass
>>>>>>>>>>> this
>>>>>>>>>>> value as
>>>>>>>>>>> a
>>>>>>>>>>> config
>>>>>>>>>>> attribute,
>>>>>>>>>>> neither
>>>>>>>>>>> the
>>>>>>>>>>> size
>>>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>> write buffer
>>>>>>>>>>> you
>>>>>>>>>>> talked
>>>>>>>>>>> about.
>>>>>>>>>>>
>>>>>>>>>>> So
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> end,
>>>>>>>>>>> is
>>>>>>>>>>> there a
>>>>>>>>>>> way
>>>>>>>>>>> with
>>>>>>>>>>> current
>>>>>>>>>>> AHC
>>>>>>>>>>> code
>>>>>>>>>>> to
>>>>>>>>>>> use
>>>>>>>>>>> this
>>>>>>>>>>> "canWrite
>>>>>>>>>>> =
>>>>>>>>>>> false"
>>>>>>>>>>> behavior?
>>>>>>>>>>>
>>>>>>>>>>> If
>>>>>>>>>>> not,
>>>>>>>>>>> can
>>>>>>>>>>> you
>>>>>>>>>>> please
>>>>>>>>>>> provide
>>>>>>>>>>> a
>>>>>>>>>>> way
>>>>>>>>>>> to
>>>>>>>>>>> set
>>>>>>>>>>> this
>>>>>>>>>>> behavior
>>>>>>>>>>> on
>>>>>>>>>>> v1.7.20
>>>>>>>>>>> ? thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> PS:
>>>>>>>>>>> does
>>>>>>>>>>> it
>>>>>>>>>>> make
>>>>>>>>>>> sens
>>>>>>>>>>> to
>>>>>>>>>>> use
>>>>>>>>>>> the
>>>>>>>>>>> same
>>>>>>>>>>> number
>>>>>>>>>>> of
>>>>>>>>>>> bytes un
>>>>>>>>>>> the
>>>>>>>>>>> feed(Buffer)
>>>>>>>>>>> method
>>>>>>>>>>> and
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>> ? do
>>>>>>>>>>> you
>>>>>>>>>>> have
>>>>>>>>>>> any
>>>>>>>>>>> tuning
>>>>>>>>>>> recommandation?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/8/29
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke <ryan.lubke_at_oracle.com
>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>
>>>>>>>>>>> Please
>>>>>>>>>>> disregard.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>
>>>>>>>>>>>> Could
>>>>>>>>>>>> you
>>>>>>>>>>>> also
>>>>>>>>>>>> provide
>>>>>>>>>>>> a
>>>>>>>>>>>> sample
>>>>>>>>>>>> of
>>>>>>>>>>>> how
>>>>>>>>>>>> you're
>>>>>>>>>>>> performing
>>>>>>>>>>>> your
>>>>>>>>>>>> feed?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> -rl
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>> Lubke
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd
>>>>>>>>>>>>> recommend
>>>>>>>>>>>>> looking
>>>>>>>>>>>>> at
>>>>>>>>>>>>> Connection.canWrite()
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> and
>>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener)
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> By
>>>>>>>>>>>>> default,
>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>> will
>>>>>>>>>>>>> configure
>>>>>>>>>>>>> the
>>>>>>>>>>>>> async
>>>>>>>>>>>>> write
>>>>>>>>>>>>> queue
>>>>>>>>>>>>> length
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> four
>>>>>>>>>>>>> times
>>>>>>>>>>>>> the
>>>>>>>>>>>>> write
>>>>>>>>>>>>> buffer
>>>>>>>>>>>>> size
>>>>>>>>>>>>> (which
>>>>>>>>>>>>> is
>>>>>>>>>>>>> based
>>>>>>>>>>>>> off
>>>>>>>>>>>>> the
>>>>>>>>>>>>> socket
>>>>>>>>>>>>> write
>>>>>>>>>>>>> buffer).
>>>>>>>>>>>>> When
>>>>>>>>>>>>> this
>>>>>>>>>>>>> queue
>>>>>>>>>>>>> exceeds
>>>>>>>>>>>>> this
>>>>>>>>>>>>> value,
>>>>>>>>>>>>> canWrite()
>>>>>>>>>>>>> will
>>>>>>>>>>>>> return
>>>>>>>>>>>>> false.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> When
>>>>>>>>>>>>> this
>>>>>>>>>>>>> occurs,
>>>>>>>>>>>>> you
>>>>>>>>>>>>> can
>>>>>>>>>>>>> register
>>>>>>>>>>>>> a WriteListener
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> notified
>>>>>>>>>>>>> when
>>>>>>>>>>>>> the
>>>>>>>>>>>>> queue
>>>>>>>>>>>>> length
>>>>>>>>>>>>> is
>>>>>>>>>>>>> below
>>>>>>>>>>>>> the
>>>>>>>>>>>>> configured
>>>>>>>>>>>>> max
>>>>>>>>>>>>> and
>>>>>>>>>>>>> then
>>>>>>>>>>>>> simulate
>>>>>>>>>>>>> blocking
>>>>>>>>>>>>> until
>>>>>>>>>>>>> the
>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>> callback
>>>>>>>>>>>>> has
>>>>>>>>>>>>> been
>>>>>>>>>>>>> invoked.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> final
>>>>>>>>>>>>> FutureImpl<Boolean>
>>>>>>>>>>>>> future
>>>>>>>>>>>>> = Futures.createSafeFuture();
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> //
>>>>>>>>>>>>> Connection
>>>>>>>>>>>>> may
>>>>>>>>>>>>> be
>>>>>>>>>>>>> obtained
>>>>>>>>>>>>> by
>>>>>>>>>>>>> calling
>>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>>
>>>>>>>>>>>>> connection.notifyCanWrite(new
>>>>>>>>>>>>> WriteHandler()
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>
>>>>>>>>>>>>> public
>>>>>>>>>>>>> void
>>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>>> throws
>>>>>>>>>>>>> Exception
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>
>>>>>>>>>>>>> public
>>>>>>>>>>>>> void
>>>>>>>>>>>>> onError(Throwable
>>>>>>>>>>>>> t)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> });
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> try
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> final
>>>>>>>>>>>>> long
>>>>>>>>>>>>> writeTimeout
>>>>>>>>>>>>> = 30;
>>>>>>>>>>>>>
>>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch
>>>>>>>>>>>>> (ExecutionException
>>>>>>>>>>>>> e)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>
>>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch
>>>>>>>>>>>>> (Exception
>>>>>>>>>>>>> e)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>>> httpCtx
>>>>>>>>>>>>> = HttpTransactionContext.get(connection);
>>>>>>>>>>>>>
>>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>> did
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> using
>>>>>>>>>>>>>> a blocking
>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> totally
>>>>>>>>>>>>>> useless
>>>>>>>>>>>>>> because
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> consuming
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> blocking
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>> never
>>>>>>>>>>>>>> blocks
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> feeding,
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> was
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>> intention
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> beginning.
>>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> depends
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>> strategy
>>>>>>>>>>>>>> used?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I use
>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>> default
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> use SameThreadIOStrategy
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>> think
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> related
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> IO
>>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> end
>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>> a 70m
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> 50m,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> put
>>>>>>>>>>>>>> a Thread.sleep(30)
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>> send
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>> connection
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> server
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> good
>>>>>>>>>>>>>> here,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> production
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> normally
>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>> better
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> far
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> I know.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>> things
>>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> work
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> block
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> too
>>>>>>>>>>>>>> much
>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> send.
>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> possible?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS:
>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>> able
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> send
>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> 100mo
>>>>>>>>>>>>>> heap,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>> a sleep
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> 100ms
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>> 100k
>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> bodygenerator
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By
>>>>>>>>>>>>>> chance
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> MessageCloner
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>> filter?
>>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>> I don't
>>>>>>>>>>>>>> really
>>>>>>>>>>>>>> understand
>>>>>>>>>>>>>> why
>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> cloned,
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>> I remove
>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>> How?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>> trying
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> send
>>>>>>>>>>>>>> a 500m
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> a heap
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>> real
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> cases
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> files
>>>>>>>>>>>>>> under
>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>> consumption
>>>>>>>>>>>>>> because
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> x parallel
>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> server
>>>>>>>>>>>>>> according
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>> activity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>> try
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> check
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> using
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>> reduced
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>> footprint
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> point
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> far
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> allocation
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> concerned,
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> untunable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That
>>>>>>>>>>>>>> said,
>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>> free
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> open
>>>>>>>>>>>>>> a feature
>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> second
>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>> size.
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>> very
>>>>>>>>>>>>>> application
>>>>>>>>>>>>>> specific.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>> curious,
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> large
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> sending?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> buffers
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> a size
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> 33842
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> limit
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> near
>>>>>>>>>>>>>>> half
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> capacity.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> tune
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> consumes
>>>>>>>>>>>>>>> less
>>>>>>>>>>>>>>> memory?
>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> ideal
>>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> reviewing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>> today,
>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>> again!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> OOM:
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> stands
>>>>>>>>>>>>>>> now,
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> SSLFilter,
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>> a buffer
>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> order
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> accommodate
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> encrypted
>>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>>> So
>>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Depending
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>> endpoints,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> becoming
>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>> We
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> situation,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>> pretty
>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>> shielded
>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>> I will
>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> leverage
>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> With
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>> usecase
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> works,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> uploaded
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>> a big
>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> body
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> that?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>> memory,
>>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>>> of:
>>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> SSLUtils$3
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> BufferBuffers
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> WriteResult
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> AsyncWriteQueueRecord
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> exemple
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>> stacktrace:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>>> idea?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Excellent!
>>>>>>>>>>>>>>>> Looking
>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>>> request!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> works
>>>>>>>>>>>>>>>>> fine,
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> a pull
>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> tomorrow
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a better
>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> Part
>>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I created
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> OutputStream
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> redirects
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>>> feeder.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>> I currently
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> feeder
>>>>>>>>>>>>>>>>> feeds
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>> polling
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>> I need
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> expose
>>>>>>>>>>>>>>>>> a limit
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> something,
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>> a thread
>>>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> filepart
>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes,
>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>> I was
>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> tackle
>>>>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>>> up
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok
>>>>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think
>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>> do,
>>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>>> bodyGenerator
>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder
>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Request
>>>>>>>>>>>>>>>>>> uploadRequest1
>>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ListenableFuture<Response>
>>>>>>>>>>>>>>>>>> asyncRes
>>>>>>>>>>>>>>>>>> = asyncHttpClient
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Response
>>>>>>>>>>>>>>>>>> uploadResponse
>>>>>>>>>>>>>>>>>> = asyncRes.get();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I guess
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> interesting
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/8/26
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>> / Grizzly
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> streaming,
>>>>>>>>>>>>>>>>>> I mean
>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>>> a byte[]
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> contans
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>> file,
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>>> a large
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>> tryied
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> a Heap
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> ByteBuffer
>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> managers,
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> reallocate=true/false
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>> success.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> content
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> appended
>>>>>>>>>>>>>>>>>> wto
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> BufferOutputStream,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> underlying
>>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>> uploading
>>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> Grizzly?
>>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>>>>> tied
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>> Multipart
>>>>>>>>>>>>>>>>>> uploads,
>>>>>>>>>>>>>>>>>> you'd
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> convert
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> multipart
>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> Buffers
>>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>> a closer
>>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> area
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Btw
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>> upload.
>>>>>>>>>>>>>>>>>> I receive
>>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> transmit
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> a storage
>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>> (like
>>>>>>>>>>>>>>>>>> S3).
>>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> bevause
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> streaming
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>>> fle
>>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>> system,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>> provides
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> input
>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>>
>>
>