Would having a different listener that will be notified once async
transferring has been started work better for you?
Something like:
onAsyncTransferInitiated() {
// invoke your feed method
}
?
Sébastien Lorber wrote:
> Unfortunatly I won't be able to use the Feeder non-blocking stuff for
> now, because of how the multipart request in handled in AHC
>
>
> Here's my feeding method:
>
> public void feed() throws IOException {
> Part[] partsArray = parts.toArray(new Part[parts.size()]);
> try ( OutputStream outputStream = createFeedingOutputStream() ) {
> Part.sendParts(outputStream,partsArray,multipartBoundary);
> } catch (Exception e) {
> throw new IllegalStateException("Unable to feed the
> FeedableBodyGenerator",e);
> }
> }
>
>
> As you can see, the multipart Parts array can only be pushed to the
> OutputStream, I don't have any way to "pull" the data when the
> canFeed() method is triggered.
>
>
> But I've seen that there's
> a com.ning.http.multipart.MultipartBody#read that seems to provide a
> memory efficient way to pull data from a Multipart body...
>
> Should see what I come up with this
>
>
> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
> <mailto:lorber.sebastien_at_gmail.com>>
>
> It seems the Feeder is highly recommended but not mandatory so I
> tried without.
>
> With my existing code it seems there is a synchronization problem.
>
>
> The feeding threads get locked to the prematureFeed.get();
>
> So the Grizzly kernel threads are unable to acquire the lock
> required to enter the initializeAsynchronousTransfer method
>
>
>
> Will try with an implementation of Feeder
>
>
>
> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
> <mailto:lorber.sebastien_at_gmail.com>>
>
> Hmmm it seems I have a problem with one of your maven plugins.
> I'll try to bypass it, but for info:
>
> ➜ ahc2 git:(ahc-1.7.x) mvn clean install
> [WARNING]
> [WARNING] Some problems were encountered while building the
> effective settings
> [WARNING]
> 'profiles.profile[default].repositories.repository.id
> <http://repositories.repository.id>' must be unique but found
> duplicate repository with id fullsix-maven-repository @
> /home/slorber/.m2/settings.xml
> [WARNING]
> [INFO] Scanning for projects...
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building Asynchronous Http Client 1.7.20-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO]
> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
> async-http-client ---
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
> (enforce-maven) @ async-http-client ---
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce
> (enforce-versions) @ async-http-client ---
> [INFO]
> [INFO] --- maven-resources-plugin:2.4.3:resources
> (default-resources) @ async-http-client ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory
> /home/slorber/Bureau/ahc2/src/main/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:2.3.2:compile
> (default-compile) @ async-http-client ---
> [INFO] Compiling 158 source files to
> /home/slorber/Bureau/ahc2/target/classes
> [INFO]
> *[INFO] --- animal-sniffer-maven-plugin:1.6:check
> (check-java-1.5-compat) @ async-http-client ---*
> *[INFO] Checking unresolved references to
> org.codehaus.mojo.signature:java15:1.0*
> *[ERROR] Undefined reference:
> java/io/IOException.<init>(Ljava/lang/Throwable;)V in
> /home/slorber/Bureau/ahc2/target/classes/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.class*
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 8.747s
> [INFO] Finished at: Tue Sep 10 11:25:41 CEST 2013
> [INFO] Final Memory: 30M/453M
> [INFO]
> ------------------------------------------------------------------------
> *[ERROR] Failed to execute goal
> org.codehaus.mojo:animal-sniffer-maven-plugin:1.6:check
> (check-java-1.5-compat) on project async-http-client:
> Signature errors found. Verify them and put
> @IgnoreJRERequirement on them. -> [Help 1]*
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run
> Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> [ERROR]
> [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>
>
>
> 2013/9/10 Sébastien Lorber <lorber.sebastien_at_gmail.com
> <mailto:lorber.sebastien_at_gmail.com>>
>
> Ok thank you, I'll try to implement that today and will
> give you my feedback :)
>
>
> 2013/9/10 Ryan Lubke <ryan.lubke_at_oracle.com
> <mailto:ryan.lubke_at_oracle.com>>
>
> Okay,
>
> I've committed my initial changes to the AHC
> repository. Here's a summary of the changes:
>
>
> *Improvements to the FeedableBodyGenerator (Grizzly's).
> - Don't allow queueing of data before
> initiateAsyncTransfer has been invoked. In low memory
> heaps, this could lead to an OOM if the source is
> feeding too fast. The new behavior is to
> block until initiateAsyncTransfer is called, at which
> time the blocked thread may proceed with
> the feed operation.
> - Introduce the concept of a Feeder. Implementations
> are responsible, at a high level, for:
> + letting the provider know that data is available to
> be fed without blocking
> + allowing the registration of a callback that the
> Feeder implementation may invoke
> to signal that more data is available, if it wasn't
> available at a previous point in time.
> - When using a Feeder with a secure request, the SSL
> handshake will be kicked off by the
> initiateAsyncTransfer call, but feeding of data will
> not occur until the handshake is complete.
> This is necessary as the SSLFilter will queue up all
> writes until the handshake is complete,
> and currently, the buffer isn't tied in with the
> transport flow control mechanism.
> NOTE: This new SSL behavior is not currently applied
> when invoking the feed() method
> outside the context of a Feeder. Still need to address
> that.
> - Exposed configuration of the async write queue limit
> through the FeedableBodyGenerator.
> This is an improvement on using a TransportCustomizer
> as any configuration there is
> transport-wide, and therefor applied to all
> Connections. By exposing it here, each feeder
> may have a different byte limit.
> - Improved documentation for this class*
>
>
>
> I recommend reading through the javadoc comments in
> the source [1] for FeedableBodyGenerator (comments
> welcome).
> Additionally, I would re-work your code to leverage
> the Feeder instead of calling feed() directly.
>
> If you have issues implementing Feeder, do let us know.
>
> If you have additional questions, again, let us know.
>
> Thanks,
> -rl
>
> [1]
> https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>
>
>
> Ryan Lubke wrote:
>>
>>
>> Sébastien Lorber wrote:
>>> So in the end I've end up with an implementation
>>> that's working for me.
>>>
>>>
>>> I think there are 2 bugs:
>>>
>>> 1) The bytes can accumulate in the
>>> FeedableBodyGenerator queue if the initialize(ctx)
>>> method is not called fast enough.
>>> This can be solved by using a BlockingQueue of size
>>> 1 and the put() method.
>>>
>>> 2) Once the context is injected, the
>>> FeedableBodyGenerator flushes the queue.
>>> The matter is that if the connection is new, not
>>> warmed up by a previous request, then the SSL
>>> handshake is not done yet, and it seems that the
>>> bytes are accumulated in some part of the SSL filter
>>> which doesn't deliver them to the connection until
>>> the handshake has completed, so c.canWrite()
>>> continues to return true.
>>>
>>>
>>>
>>>
>>> I have replaced some part of the
>>> FeedableBodyGenerator to test this and it works
>>> pretty fine. See what I have changed:
>>>
>>> 1)
>>> private final BlockingQueue<BodyPart> queue =
>>> new LinkedBlockingQueue<BodyPart>(1);
>>>
>>>
>>> 2)
>>> public void feed(final Buffer buffer, final
>>> boolean last) throws IOException {
>>> try {
>>> queue.put(new BodyPart(buffer, last));
>>> } catch (InterruptedException e) {
>>> throw new RuntimeException(e);
>>> }
>>> queueSize.incrementAndGet();
>>> if (context != null) {
>>> blockUntilConnectionIsReadyToWrite(context);
>>> flushQueue(true);
>>> }
>>> }
>>>
>>> private void
>>> blockUntilConnectionIsReadyToWrite(FilterChainContext fcc)
>>> {
>>> if ( !connectionIsReadyToWrite(fcc) ) {
>>> while ( !connectionIsReadyToWrite(fcc) ) {
>>> try { Thread.sleep(10); } catch (
>>> Exception e ) { throw new RuntimeException(e); }
>>> }
>>> }
>>> }
>>>
>>> private boolean
>>> connectionIsReadyToWrite(FilterChainContext fcc) {
>>> Connection connection = fcc.getConnection();
>>> SSLEngine sslEngine =
>>> SSLUtils.getSSLEngine(connection);
>>> return sslEngine != null &&
>>> !SSLUtils.isHandshaking(sslEngine);
>>> }
>>>
>>>
>>>
>>> What do you think?
>>
>> We had come to similar conclusions on this end. I'm
>> still working through testing the idea I mentioned
>> previously (took longer than I expected - sorry).
>> I hope to have something for you to test very soon.
>>
>> Note that it will be taking the above into account as
>> well.
>>>
>>>
>>>
>>>
>>> 2013/9/5 Sébastien Lorber
>>> <lorber.sebastien_at_gmail.com
>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>
>>>
>>> I have tried to put a while ( context == null )
>>> Thread.sleep but it doesn't seem to work, when
>>> the context gets injected, after the sleeps,
>>> there's an OOM
>>>
>>> So I hope you'll have more success with your
>>> alternative :)
>>>
>>>
>>>
>>> I have done another test, remember my code that
>>> worked, which previously "warmed" the Thread
>>> with an useless request.
>>>
>>> private Runnable uploadSingleDocumentRunnable
>>> = new Runnable() {
>>> @Override
>>> public void run() {
>>> try {
>>> getUselessSessionCode();
>>> Thread.sleep(X);
>>> uploadSingleDocument();
>>> } catch ( Exception e ) {
>>> throw new RuntimeException("file upload
>>> failed",e);
>>> }
>>> }
>>> };
>>>
>>> I have put a sleep of X between the useless
>>> warmup request, and the real upload request
>>>
>>>
>>> What I noticed is that there is a very different
>>> behavior according to the value of X
>>>
>>>
>>> Under 10 seconds, it seems the stuff is still
>>> warm, I can upload the documents.
>>> Around 10 seconds I get a stack which seems to
>>> be "connection closed" or something
>>> Above 10 seconds, I get OOM like if the stuff
>>> wasn't warm.
>>>
>>>
>>> The stacks I get for 10 seconds looks like
>>>
>>> Caused by: javax.net.ssl.SSLException: SSLEngine
>>> is CLOSED
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:295)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:133)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:95)
>>> ~[async-http-client-1.7.20-SNAPSHOT.jar:na]
>>>
>>> I think I got some other different stacks saying
>>> Connection Closed Remotely or something like that.
>>>
>>>
>>>
>>>
>>> So it seems that something is bound to my
>>> thread, and it stays bound to it for about 10
>>> seconds, do you have any idea what it could be?
>>> (My connection timeout setting seems to have no
>>> effect on this 10s threshold)
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/9/5 Ryan Lubke <ryan.lubke_at_oracle.com
>>> <mailto:ryan.lubke_at_oracle.com>>
>>>
>>> That is one solution. I'm working out an
>>> alternative right now. Stay tuned!
>>>
>>>>
>>>> Anyway it's not a problem, I think the
>>>> FeedableBodyGenerator.feed() method just
>>>> has to block until a context has been (and
>>>> ThreadCache initialized) to avoid OOM errors
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/5 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>>
>>>> What is very strange is that I tested
>>>> with/without the same sessionCode with
>>>> our previous code, the one not using
>>>> FeedableBodyGenerator, which has a high
>>>> memory consumption.
>>>> Despites the fact it had high memory
>>>> consumption, it seems work fine to
>>>> upload multiple documents if allocated
>>>> with a large heap, and the sessionCode
>>>> seems to have no effect.
>>>>
>>>> On the new impl using the
>>>> FeedableBodyGenerator, the sessionCode
>>>> sent as a multipart bodypart seems to
>>>> have an effect.
>>>>
>>>> I have tried to feed the queue before
>>>> sending the request to AHC, but this
>>>> leads to this exception (with/without
>>>> sessionCode switching)
>>>> Caused by:
>>>> java.util.concurrent.TimeoutException:
>>>> Timeout exceeded
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
>>>> at
>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
>>>> at
>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
>>>> at
>>>> org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
>>>> at
>>>> org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/5 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>> By the way, by using a low timeout
>>>> with the same sessioncode, I got
>>>> the following NPE:
>>>>
>>>> Caused by:
>>>> java.lang.NullPointerException
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>> at
>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>
>>>>
>>>> GrizzlyAsyncHttpProvider.HttpTransactionContext
>>>> httpCtx =
>>>>
>>>> getHttpTransactionContext(c);
>>>>
>>>> httpCtx.abort(e.getCause());
>>>>
>>>> I guess the httpCtx is not already
>>>> available to be aborted
>>>>
>>>>
>>>> 2013/9/5 Sébastien Lorber
>>>> <lorber.sebastien_at_gmail.com
>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>
>>>>
>>>>
>>>> This is right, here's a log I
>>>> have when I use the same
>>>> session code, ie the remote
>>>> host is blocking the data or
>>>> something.
>>>> This is obtained by running 5
>>>> parallel uploads.
>>>>
>>>> *Flushing queue of size 0 with
>>>> allowBlocking = false*
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> *Flushing queue of size 97 with
>>>> allowBlocking = false*
>>>> *Flushing queue of size 100
>>>> with allowBlocking = false*
>>>> *Flushing queue of size 160
>>>> with allowBlocking = true*
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 0 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> java.lang.OutOfMemoryError: GC
>>>> overhead limit exceeded
>>>> Dumping heap to
>>>> /ome/lorber/ureau/om ...
>>>> Unable to create
>>>> /ome/lorber/ureau/om: Le
>>>> fichier existe
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Disconnected from the target
>>>> VM, address: '127.0.0.1:49268
>>>> <http://127.0.0.1:49268>',
>>>> transport: 'socket'
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Otherwise, with different
>>>> session codes, I get the following:
>>>>
>>>> *Flushing queue of size 0 with
>>>> allowBlocking = false*
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> *Flushing queue of size 0 with
>>>> allowBlocking = false*
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> *Flushing queue of size 0 with
>>>> allowBlocking = false*
>>>> *Flushing queue of size 0 with
>>>> allowBlocking = false*
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> Flushing queue of size 1 with
>>>> allowBlocking = true
>>>> ... and this continues without OOM
>>>>
>>>>
>>>> So, this seems to be the problem.
>>>>
>>>>
>>>>
>>>> I think it would be great to be
>>>> able to be able to choose the
>>>> queue impl behind
>>>> that FeedableBodyGenerator,
>>>> like I suggested in my pull
>>>> request.
>>>>
>>>> See here:
>>>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>>>
>>>> Using a LinkedBlockingQueue
>>>> seems to be a nice idea in this
>>>> context, and in my case I would
>>>> probably use a queue of size 1
>>>>
>>>>
>>>> This would handle the blocking
>>>> of the feed method, without
>>>> having to use this:
>>>> if (context != null) {
>>>> flushQueue(true);
>>>> }
>>>>
>>>>
>>>> Or perhaps the feed() method
>>>> have to wait until a context is
>>>> set in the BodyGenerator ?
>>>>
>>>>
>>>> I think it would be more clear
>>>> if
>>>> the initializeAsynchronousTransfer
>>>> simply didn't flush the queue
>>>> but just setup the context.
>>>> Then the feed method would
>>>> block until there's a context
>>>> set, and then flush the queue
>>>> with blocking behavior.
>>>>
>>>> This is probably the next step,
>>>> but as we are using AHC for
>>>> async, it would probably be
>>>> great if that blocking feed()
>>>> method was called in a worker
>>>> thread instead of our main thread.
>>>> I won't use this but someone
>>>> who really wants a non-blocking
>>>> impl of performant multipart
>>>> fileupload would probably need
>>>> this, or will use an
>>>> ExecutorService for the feeding
>>>> operations as a workaround.
>>>>
>>>>
>>>>
>>>> Thanks again for your reactivity
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/9/4 Ryan Lubke
>>>> <ryan.lubke_at_oracle.com
>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>
>>>>
>>>>
>>>> Sébastien Lorber wrote:
>>>>> I've integrated this
>>>>> change and it works fine
>>>>> except a little detail.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I'm uploading files to a
>>>>> third party API (a bit
>>>>> like S3).
>>>>> This API requires a
>>>>> "sessionCode" in each
>>>>> request. So there is a
>>>>> multipart StringPart with
>>>>> that SessionCode.
>>>>>
>>>>> We used to have a cache
>>>>> which holds the
>>>>> sessionCode 30min per user
>>>>> so that we do not need to
>>>>> init a new session each time.
>>>>>
>>>>> I had troubles in this
>>>>> very specific case: when I
>>>>> upload 5 docs with the
>>>>> same session code.
>>>>> When I remove the cache
>>>>> and use 5 different
>>>>> session codes, it works fine.
>>>>>
>>>>> I guess the remote service
>>>>> is blocking concurrent
>>>>> uploads with the same
>>>>> session code. I don't know
>>>>> at all.
>>>>>
>>>>>
>>>>>
>>>>> Where I want to go is that
>>>>> I wouldn't have expected
>>>>> Grizzly to OOM
>>>>>
>>>>>
>>>>> Avertissement: Exception
>>>>> during FilterChain execution
>>>>> java.lang.OutOfMemoryError: Java
>>>>> heap space
>>>>> at
>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>> at
>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>> at
>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>>>> at
>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>> at
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>> at
>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>> at
>>>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>>>> at
>>>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Caused by:
>>>>> java.util.concurrent.TimeoutException:
>>>>> null
>>>>> at
>>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>> at
>>>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>> at
>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>>>
>>>>>
>>>>>
>>>>> multipart.body.generator.feeder.buffer=100000
>>>>> -> size of each Buffer
>>>>> sent to the
>>>>> FeedableBodyGenerator
>>>>> transport.max.pending.bytes=1000000
>>>>> (I tried other settings,
>>>>> including AUTO_SIZE)
>>>>>
>>>>>
>>>>>
>>>>> Do you have any idea why
>>>>> is there an OOM with these
>>>>> settings?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Perhaps it is because the
>>>>> feed() method of
>>>>> FeedableBodyGenerator
>>>>> doesn't block until the
>>>>> context is initialized.
>>>>> I guess
>>>>> the initializeAsynchronousTransfer
>>>>> is called only once the
>>>>> connection is established,
>>>>> and perhaps a lot of
>>>>> Buffer are added to the
>>>>> queue...
>>>> Yes, it's invoked once the
>>>> request has been
>>>> dispatched, so if the
>>>> generator is fed a lot
>>>> before the request, I could
>>>> see this happening.
>>>> I'll see what I can do to
>>>> alleviate that case.
>>>>
>>>>>
>>>>> But I'm not sure at all
>>>>> because the session code
>>>>> is transmitted as a
>>>>> BodyPart and I get the
>>>>> same problem if i put it
>>>>> as the first or last
>>>>> multipart.
>>>>>
>>>>> It's not a big deal,
>>>>> perhaps I should always
>>>>> use a different session
>>>>> code for concurrent
>>>>> operations but I'd like to
>>>>> be sure that we won't have
>>>>> this issue in production...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/9/3 Ryan Lubke
>>>>> <ryan.lubke_at_oracle.com
>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>
>>>>> Good catch. Fixed.
>>>>>
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> There's a little
>>>>>> mistake in the
>>>>>> grizzly ahc provider
>>>>>> relative to the write
>>>>>> queue size.
>>>>>>
>>>>>>
>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>>>
>>>>>> As you can see, the
>>>>>> TransportCustomizer
>>>>>> is called, and then
>>>>>> the write queue size
>>>>>> (among other things)
>>>>>> is set to AUTO_SIZE
>>>>>> (instead of
>>>>>> previously UNLIMITED)
>>>>>>
>>>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I think the default
>>>>>> settings like this
>>>>>> AUTO_SIZE attribute
>>>>>> should be set before
>>>>>> the customization of
>>>>>> the transport, or
>>>>>> they would override
>>>>>> the value we customized.
>>>>>>
>>>>>>
>>>>>> This is actually my
>>>>>> case, since I can't
>>>>>> reproduce my "bug"
>>>>>> which is "high memory
>>>>>> consumption", even
>>>>>> when using -1 /
>>>>>> UNLIMITED in the
>>>>>> TransportCustomizer.
>>>>>>
>>>>>>
>>>>>> This could work fine
>>>>>> for me with
>>>>>> AUTO_SIZE, but I'd
>>>>>> rather be able to
>>>>>> tune this parameter
>>>>>> during load tests to
>>>>>> see the effect.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/8/31 Sebastien
>>>>>> Lorber
>>>>>> <lorber.sebastien_at_gmail.com
>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>
>>>>>> Thanks i will
>>>>>> ckeck that on
>>>>>> monday. I can now
>>>>>> upload a 500m
>>>>>> file with 40m
>>>>>> heap size ;)
>>>>>>
>>>>>> Envoyé de mon iPhone
>>>>>>
>>>>>> Le 30 août 2013 à
>>>>>> 20:49, Ryan Lubke
>>>>>> <ryan.lubke_at_oracle.com
>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>> a écrit :
>>>>>>
>>>>>>> I'm going to be
>>>>>>> updating the
>>>>>>> Grizzly provider
>>>>>>> such that
>>>>>>> AUTO_SIZE (not
>>>>>>> AUTO_TUNE) is
>>>>>>> the default, so
>>>>>>> you can avoid
>>>>>>> the use of the
>>>>>>> TransportCustomizer.
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>> Regarding your
>>>>>>>> tuning
>>>>>>>> question, I
>>>>>>>> would probably
>>>>>>>> set the value
>>>>>>>> to
>>>>>>>> AsyncQueueWriter.AUTO_TUNE
>>>>>>>> (this will be
>>>>>>>> four times the
>>>>>>>> socket write
>>>>>>>> buffer) and see
>>>>>>>> how that works.
>>>>>>>>
>>>>>>>> Ryan Lubke wrote:
>>>>>>>>> A question
>>>>>>>>> first. With
>>>>>>>>> these changes,
>>>>>>>>> your memory
>>>>>>>>> usage is more
>>>>>>>>> inline with
>>>>>>>>> what you were
>>>>>>>>> looking for?
>>>>>>>>>
>>>>>>>>> Sébastien
>>>>>>>>> Lorber wrote:
>>>>>>>>>> By the way,
>>>>>>>>>> do you have
>>>>>>>>>> any idea when
>>>>>>>>>> the 1.7.20
>>>>>>>>>> will be
>>>>>>>>>> released
>>>>>>>>>> (with these
>>>>>>>>>> new
>>>>>>>>>> improvements?)
>>>>>>>>>>
>>>>>>>>>> We would like
>>>>>>>>>> to know if we
>>>>>>>>>> wait for a
>>>>>>>>>> release or if
>>>>>>>>>> we install
>>>>>>>>>> our own temp
>>>>>>>>>> release on
>>>>>>>>>> Nexus :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/30
>>>>>>>>>> Sébastien
>>>>>>>>>> Lorber
>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>
>>>>>>>>>> Thank
>>>>>>>>>> you, it
>>>>>>>>>> works fine!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I just
>>>>>>>>>> had to
>>>>>>>>>> modify a
>>>>>>>>>> single
>>>>>>>>>> line
>>>>>>>>>> after
>>>>>>>>>> your commit.
>>>>>>>>>>
>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>>>>>> ->
>>>>>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If I let
>>>>>>>>>> the
>>>>>>>>>> initial
>>>>>>>>>> value
>>>>>>>>>> (-1) it
>>>>>>>>>> won't
>>>>>>>>>> block,
>>>>>>>>>> canWrite
>>>>>>>>>> always
>>>>>>>>>> returns true
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Btw, on
>>>>>>>>>> AHC I
>>>>>>>>>> didn't
>>>>>>>>>> find any
>>>>>>>>>> way to
>>>>>>>>>> pass this
>>>>>>>>>> value as
>>>>>>>>>> a config
>>>>>>>>>> attribute, neither
>>>>>>>>>> the size
>>>>>>>>>> of the
>>>>>>>>>> write
>>>>>>>>>> buffer
>>>>>>>>>> you
>>>>>>>>>> talked about.
>>>>>>>>>>
>>>>>>>>>> So in the
>>>>>>>>>> end, is
>>>>>>>>>> there a
>>>>>>>>>> way with
>>>>>>>>>> current
>>>>>>>>>> AHC code
>>>>>>>>>> to use
>>>>>>>>>> this
>>>>>>>>>> "canWrite
>>>>>>>>>> = false"
>>>>>>>>>> behavior?
>>>>>>>>>> If not,
>>>>>>>>>> can you
>>>>>>>>>> please
>>>>>>>>>> provide a
>>>>>>>>>> way to
>>>>>>>>>> set this
>>>>>>>>>> behavior
>>>>>>>>>> on
>>>>>>>>>> v1.7.20 ?
>>>>>>>>>> thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> PS: does
>>>>>>>>>> it make
>>>>>>>>>> sens to
>>>>>>>>>> use the
>>>>>>>>>> same
>>>>>>>>>> number of
>>>>>>>>>> bytes un
>>>>>>>>>> the
>>>>>>>>>> feed(Buffer)
>>>>>>>>>> method
>>>>>>>>>> and in
>>>>>>>>>> the
>>>>>>>>>> setMaxPendingBytesPerConnection(10000);
>>>>>>>>>> ? do you
>>>>>>>>>> have any
>>>>>>>>>> tuning
>>>>>>>>>> recommandation?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/29
>>>>>>>>>> Ryan
>>>>>>>>>> Lubke
>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>
>>>>>>>>>> Please disregard.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ryan
>>>>>>>>>> Lubke
>>>>>>>>>> wrote:
>>>>>>>>>>> Sébastien,
>>>>>>>>>>>
>>>>>>>>>>> Could you
>>>>>>>>>>> also
>>>>>>>>>>> provide
>>>>>>>>>>> a
>>>>>>>>>>> sample
>>>>>>>>>>> of
>>>>>>>>>>> how
>>>>>>>>>>> you're
>>>>>>>>>>> performing
>>>>>>>>>>> your
>>>>>>>>>>> feed?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -rl
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> Lubke wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sébastien,
>>>>>>>>>>>>
>>>>>>>>>>>> I'd
>>>>>>>>>>>> recommend
>>>>>>>>>>>> looking
>>>>>>>>>>>> at
>>>>>>>>>>>> Connection.canWrite()
>>>>>>>>>>>> [1]
>>>>>>>>>>>> and
>>>>>>>>>>>> Connection.notifyCanWrite(WriteListener)
>>>>>>>>>>>> [1]
>>>>>>>>>>>>
>>>>>>>>>>>> By
>>>>>>>>>>>> default,
>>>>>>>>>>>> Grizzly
>>>>>>>>>>>> will configure
>>>>>>>>>>>> the
>>>>>>>>>>>> async
>>>>>>>>>>>> write
>>>>>>>>>>>> queue
>>>>>>>>>>>> length
>>>>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> four times
>>>>>>>>>>>> the
>>>>>>>>>>>> write
>>>>>>>>>>>> buffer
>>>>>>>>>>>> size (which
>>>>>>>>>>>> is
>>>>>>>>>>>> based
>>>>>>>>>>>> off
>>>>>>>>>>>> the
>>>>>>>>>>>> socket
>>>>>>>>>>>> write
>>>>>>>>>>>> buffer).
>>>>>>>>>>>> When this
>>>>>>>>>>>> queue
>>>>>>>>>>>> exceeds
>>>>>>>>>>>> this value,
>>>>>>>>>>>> canWrite()
>>>>>>>>>>>> will return
>>>>>>>>>>>> false.
>>>>>>>>>>>>
>>>>>>>>>>>> When this
>>>>>>>>>>>> occurs,
>>>>>>>>>>>> you
>>>>>>>>>>>> can
>>>>>>>>>>>> register
>>>>>>>>>>>> a
>>>>>>>>>>>> WriteListener
>>>>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> notified
>>>>>>>>>>>> when the
>>>>>>>>>>>> queue
>>>>>>>>>>>> length
>>>>>>>>>>>> is
>>>>>>>>>>>> below
>>>>>>>>>>>> the
>>>>>>>>>>>> configured
>>>>>>>>>>>> max
>>>>>>>>>>>> and
>>>>>>>>>>>> then simulate
>>>>>>>>>>>> blocking
>>>>>>>>>>>> until
>>>>>>>>>>>> the
>>>>>>>>>>>> onWritePossible()
>>>>>>>>>>>> callback
>>>>>>>>>>>> has
>>>>>>>>>>>> been invoked.
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> final
>>>>>>>>>>>> FutureImpl<Boolean>
>>>>>>>>>>>> future
>>>>>>>>>>>> =
>>>>>>>>>>>> Futures.createSafeFuture();
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> //
>>>>>>>>>>>> Connection
>>>>>>>>>>>> may
>>>>>>>>>>>> be
>>>>>>>>>>>> obtained
>>>>>>>>>>>> by
>>>>>>>>>>>> calling
>>>>>>>>>>>> FilterChainContext.getConnection().
>>>>>>>>>>>>
>>>>>>>>>>>> connection.notifyCanWrite(new
>>>>>>>>>>>> WriteHandler()
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> @Override
>>>>>>>>>>>>
>>>>>>>>>>>> public
>>>>>>>>>>>> void onWritePossible()
>>>>>>>>>>>> throws
>>>>>>>>>>>> Exception
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> future.result(Boolean.TRUE);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> @Override
>>>>>>>>>>>>
>>>>>>>>>>>> public
>>>>>>>>>>>> void onError(Throwable
>>>>>>>>>>>> t) {
>>>>>>>>>>>>
>>>>>>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> });
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> final
>>>>>>>>>>>> long writeTimeout
>>>>>>>>>>>> = 30;
>>>>>>>>>>>>
>>>>>>>>>>>> future.get(writeTimeout,
>>>>>>>>>>>> TimeUnit.MILLISECONDS);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>> catch
>>>>>>>>>>>> (ExecutionException
>>>>>>>>>>>> e) {
>>>>>>>>>>>>
>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>> httpCtx
>>>>>>>>>>>> =
>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>
>>>>>>>>>>>> httpCtx.abort(e.getCause());
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>> catch
>>>>>>>>>>>> (Exception
>>>>>>>>>>>> e) {
>>>>>>>>>>>> HttpTransactionContext
>>>>>>>>>>>> httpCtx
>>>>>>>>>>>> =
>>>>>>>>>>>> HttpTransactionContext.get(connection);
>>>>>>>>>>>>
>>>>>>>>>>>> httpCtx.abort(e);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html.
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien
>>>>>>>>>>>> Lorber
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Ryan,
>>>>>>>>>>>>> I've
>>>>>>>>>>>>> did some
>>>>>>>>>>>>> other
>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It
>>>>>>>>>>>>> seems
>>>>>>>>>>>>> that
>>>>>>>>>>>>> using
>>>>>>>>>>>>> a
>>>>>>>>>>>>> blocking
>>>>>>>>>>>>> queue
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the FeedableBodyGenerator
>>>>>>>>>>>>> is
>>>>>>>>>>>>> totally
>>>>>>>>>>>>> useless
>>>>>>>>>>>>> because
>>>>>>>>>>>>> the thread
>>>>>>>>>>>>> consuming
>>>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>> not blocking
>>>>>>>>>>>>> and the
>>>>>>>>>>>>> queue
>>>>>>>>>>>>> never
>>>>>>>>>>>>> blocks
>>>>>>>>>>>>> the feeding,
>>>>>>>>>>>>> which
>>>>>>>>>>>>> was my
>>>>>>>>>>>>> intention
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the beginning.
>>>>>>>>>>>>> Maybe
>>>>>>>>>>>>> it
>>>>>>>>>>>>> depends
>>>>>>>>>>>>> on
>>>>>>>>>>>>> the IO
>>>>>>>>>>>>> strategy
>>>>>>>>>>>>> used?
>>>>>>>>>>>>> I
>>>>>>>>>>>>> use AHC
>>>>>>>>>>>>> default
>>>>>>>>>>>>> which
>>>>>>>>>>>>> seems
>>>>>>>>>>>>> to
>>>>>>>>>>>>> use SameThreadIOStrategy
>>>>>>>>>>>>> so
>>>>>>>>>>>>> I
>>>>>>>>>>>>> don't
>>>>>>>>>>>>> think
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> related
>>>>>>>>>>>>> to
>>>>>>>>>>>>> the IO
>>>>>>>>>>>>> strategy.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> So
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the end
>>>>>>>>>>>>> I
>>>>>>>>>>>>> can upload
>>>>>>>>>>>>> a
>>>>>>>>>>>>> 70m file
>>>>>>>>>>>>> with
>>>>>>>>>>>>> a
>>>>>>>>>>>>> heap
>>>>>>>>>>>>> of
>>>>>>>>>>>>> 50m,
>>>>>>>>>>>>> but I
>>>>>>>>>>>>> have
>>>>>>>>>>>>> to
>>>>>>>>>>>>> put a
>>>>>>>>>>>>> Thread.sleep(30)
>>>>>>>>>>>>> between
>>>>>>>>>>>>> each
>>>>>>>>>>>>> 100k
>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>> send
>>>>>>>>>>>>> to
>>>>>>>>>>>>> the FeedableBodyGenerator
>>>>>>>>>>>>>
>>>>>>>>>>>>> The connection
>>>>>>>>>>>>> with
>>>>>>>>>>>>> the server
>>>>>>>>>>>>> is
>>>>>>>>>>>>> not good
>>>>>>>>>>>>> here,
>>>>>>>>>>>>> but in
>>>>>>>>>>>>> production
>>>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>> normally
>>>>>>>>>>>>> a
>>>>>>>>>>>>> lot better
>>>>>>>>>>>>> as
>>>>>>>>>>>>> far as
>>>>>>>>>>>>> I
>>>>>>>>>>>>> know.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've
>>>>>>>>>>>>> tried
>>>>>>>>>>>>> things
>>>>>>>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>>>>>>>> but it
>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>> seem
>>>>>>>>>>>>> to
>>>>>>>>>>>>> work
>>>>>>>>>>>>> for me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like
>>>>>>>>>>>>> the Grizzly
>>>>>>>>>>>>> internals
>>>>>>>>>>>>> to
>>>>>>>>>>>>> block
>>>>>>>>>>>>> when
>>>>>>>>>>>>> there
>>>>>>>>>>>>> are too
>>>>>>>>>>>>> much
>>>>>>>>>>>>> pending
>>>>>>>>>>>>> bytes
>>>>>>>>>>>>> to
>>>>>>>>>>>>> send.
>>>>>>>>>>>>> Is
>>>>>>>>>>>>> it
>>>>>>>>>>>>> possible?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> PS: I've
>>>>>>>>>>>>> just
>>>>>>>>>>>>> been
>>>>>>>>>>>>> able
>>>>>>>>>>>>> to
>>>>>>>>>>>>> send
>>>>>>>>>>>>> a
>>>>>>>>>>>>> 500mo
>>>>>>>>>>>>> file
>>>>>>>>>>>>> with
>>>>>>>>>>>>> 100mo
>>>>>>>>>>>>> heap,
>>>>>>>>>>>>> but it
>>>>>>>>>>>>> needed
>>>>>>>>>>>>> a
>>>>>>>>>>>>> sleep
>>>>>>>>>>>>> of
>>>>>>>>>>>>> 100ms
>>>>>>>>>>>>> between
>>>>>>>>>>>>> each
>>>>>>>>>>>>> 100k
>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>> sent
>>>>>>>>>>>>> to
>>>>>>>>>>>>> the bodygenerator
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> By
>>>>>>>>>>>>> chance
>>>>>>>>>>>>> do
>>>>>>>>>>>>> you
>>>>>>>>>>>>> if
>>>>>>>>>>>>> I can
>>>>>>>>>>>>> remove
>>>>>>>>>>>>> the
>>>>>>>>>>>>> MessageCloner
>>>>>>>>>>>>> used
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SSL
>>>>>>>>>>>>> filter?
>>>>>>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>>>>>>
>>>>>>>>>>>>> It
>>>>>>>>>>>>> seems
>>>>>>>>>>>>> to
>>>>>>>>>>>>> allocate
>>>>>>>>>>>>> a lot
>>>>>>>>>>>>> of
>>>>>>>>>>>>> memory.
>>>>>>>>>>>>> I don't
>>>>>>>>>>>>> really
>>>>>>>>>>>>> understand
>>>>>>>>>>>>> why
>>>>>>>>>>>>> messages
>>>>>>>>>>>>> have
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> cloned,
>>>>>>>>>>>>> can
>>>>>>>>>>>>> I remove
>>>>>>>>>>>>> this?
>>>>>>>>>>>>> How?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/29
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> <lorber.sebastien_at_gmail.com
>>>>>>>>>>>>> <mailto:lorber.sebastien_at_gmail.com>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm
>>>>>>>>>>>>> trying
>>>>>>>>>>>>> to
>>>>>>>>>>>>> send
>>>>>>>>>>>>> a 500m
>>>>>>>>>>>>> file
>>>>>>>>>>>>> for
>>>>>>>>>>>>> my
>>>>>>>>>>>>> tests
>>>>>>>>>>>>> with
>>>>>>>>>>>>> a heap
>>>>>>>>>>>>> of
>>>>>>>>>>>>> 400m.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In
>>>>>>>>>>>>> our
>>>>>>>>>>>>> real
>>>>>>>>>>>>> use
>>>>>>>>>>>>> cases
>>>>>>>>>>>>> we
>>>>>>>>>>>>> would
>>>>>>>>>>>>> probably
>>>>>>>>>>>>> have
>>>>>>>>>>>>> files
>>>>>>>>>>>>> under
>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>> but
>>>>>>>>>>>>> we
>>>>>>>>>>>>> want
>>>>>>>>>>>>> to
>>>>>>>>>>>>> reduce
>>>>>>>>>>>>> the
>>>>>>>>>>>>> memory
>>>>>>>>>>>>> consumption
>>>>>>>>>>>>> because
>>>>>>>>>>>>> we
>>>>>>>>>>>>> can
>>>>>>>>>>>>> have
>>>>>>>>>>>>> x parallel
>>>>>>>>>>>>> uploads
>>>>>>>>>>>>> on
>>>>>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>> server
>>>>>>>>>>>>> according
>>>>>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> user
>>>>>>>>>>>>> activity.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll
>>>>>>>>>>>>> try
>>>>>>>>>>>>> to
>>>>>>>>>>>>> check
>>>>>>>>>>>>> if
>>>>>>>>>>>>> using
>>>>>>>>>>>>> this
>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>> reduced
>>>>>>>>>>>>> the
>>>>>>>>>>>>> memory
>>>>>>>>>>>>> footprint
>>>>>>>>>>>>> or
>>>>>>>>>>>>> if
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> almost
>>>>>>>>>>>>> like
>>>>>>>>>>>>> before.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> At
>>>>>>>>>>>>> this
>>>>>>>>>>>>> point
>>>>>>>>>>>>> in
>>>>>>>>>>>>> time,
>>>>>>>>>>>>> as
>>>>>>>>>>>>> far
>>>>>>>>>>>>> as
>>>>>>>>>>>>> the
>>>>>>>>>>>>> SSL
>>>>>>>>>>>>> buffer
>>>>>>>>>>>>> allocation
>>>>>>>>>>>>> is
>>>>>>>>>>>>> concerned,
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> untunable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That
>>>>>>>>>>>>> said,
>>>>>>>>>>>>> feel
>>>>>>>>>>>>> free
>>>>>>>>>>>>> to
>>>>>>>>>>>>> open
>>>>>>>>>>>>> a feature
>>>>>>>>>>>>> request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As
>>>>>>>>>>>>> to
>>>>>>>>>>>>> your
>>>>>>>>>>>>> second
>>>>>>>>>>>>> question,
>>>>>>>>>>>>> there
>>>>>>>>>>>>> is
>>>>>>>>>>>>> no
>>>>>>>>>>>>> suggested
>>>>>>>>>>>>> size.
>>>>>>>>>>>>> This
>>>>>>>>>>>>> is
>>>>>>>>>>>>> all
>>>>>>>>>>>>> very
>>>>>>>>>>>>> application
>>>>>>>>>>>>> specific.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm
>>>>>>>>>>>>> curious,
>>>>>>>>>>>>> how
>>>>>>>>>>>>> large
>>>>>>>>>>>>> of
>>>>>>>>>>>>> a file
>>>>>>>>>>>>> are
>>>>>>>>>>>>> you
>>>>>>>>>>>>> sending?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> buffers
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> a size
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> 33842
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> limit
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> near
>>>>>>>>>>>>>> half
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> capacity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> tune
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> size
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> consumes
>>>>>>>>>>>>>> less
>>>>>>>>>>>>>> memory?
>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>> ideal
>>>>>>>>>>>>>> Buffer
>>>>>>>>>>>>>> size
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> send
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>> method?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/8/28
>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> reviewing
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>> today,
>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>> again!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> OOM:
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> stands
>>>>>>>>>>>>>> now,
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> SSLFilter,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>> a buffer
>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> size
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> order
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> accommodate
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> encrypted
>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>> So
>>>>>>>>>>>>>> there's
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>> increase.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Depending
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> both
>>>>>>>>>>>>>> endpoints,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>> data,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> write
>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> becoming
>>>>>>>>>>>>>> too
>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>> We
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> situation,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>> pretty
>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>> internals
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>> shielded
>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>> I will
>>>>>>>>>>>>>> see
>>>>>>>>>>>>>> what
>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>> users
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> leverage
>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With
>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>> usecase
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> works,
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> uploaded
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>> a big
>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> body
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>> that?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>> memory,
>>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>>> a lot
>>>>>>>>>>>>>>> of:
>>>>>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> SSLUtils$3
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> BufferBuffers
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> WriteResult
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> hold
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> AsyncWriteQueueRecord
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> exemple
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>> stacktrace:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError:
>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>> idea?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Excellent!
>>>>>>>>>>>>>>> Looking
>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> pull
>>>>>>>>>>>>>>> request!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> works
>>>>>>>>>>>>>>>> fine,
>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>> a pull
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>> tomorrow
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> a better
>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> Part
>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>> exist.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I created
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> OutputStream
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> redirects
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> BodyGenerator
>>>>>>>>>>>>>>>> feeder.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>> I currently
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> feeder
>>>>>>>>>>>>>>>> feeds
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> faster
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> async
>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>> polling
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>> I need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> expose
>>>>>>>>>>>>>>>> a limit
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> queue
>>>>>>>>>>>>>>>> size
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> something,
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> a thread
>>>>>>>>>>>>>>>> sleep
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>>> down
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> filepart
>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/8/27
>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes,
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>> I was
>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> tackle
>>>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> today.
>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>> follow
>>>>>>>>>>>>>>>> up
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ok
>>>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think
>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>> do,
>>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> that:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> FeedableBodyGenerator
>>>>>>>>>>>>>>>>> bodyGenerator
>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>> bodyGeneratorFeeder
>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Request
>>>>>>>>>>>>>>>>> uploadRequest1
>>>>>>>>>>>>>>>>> = new
>>>>>>>>>>>>>>>>> RequestBuilder("POST")
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ListenableFuture<Response>
>>>>>>>>>>>>>>>>> asyncRes
>>>>>>>>>>>>>>>>> = asyncHttpClient
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> .execute(new
>>>>>>>>>>>>>>>>> AsyncCompletionHandlerBase());
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Response
>>>>>>>>>>>>>>>>> uploadResponse
>>>>>>>>>>>>>>>>> = asyncRes.get();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I guess
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> interesting
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> MultipartBodyGeneratorFeeder
>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> Grizzly
>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> people
>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/8/26
>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> Lubke
>>>>>>>>>>>>>>>>> <ryan.lubke_at_oracle.com
>>>>>>>>>>>>>>>>> <mailto:ryan.lubke_at_oracle.com>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sébastien
>>>>>>>>>>>>>>>>> Lorber
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> / Grizzly
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> streaming,
>>>>>>>>>>>>>>>>> I mean
>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> allocate
>>>>>>>>>>>>>>>>> a byte[]
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> contans
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>> file,
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> OOM
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>> upload
>>>>>>>>>>>>>>>>> a large
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> tryied
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a Heap
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> ByteBuffer
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> managers,
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> reallocate=true/false
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>> success.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> whole
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> content
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> appended
>>>>>>>>>>>>>>>>> wto
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> BufferOutputStream,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> underlying
>>>>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> written.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At
>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> integration:
>>>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> a way
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>>>>> AHC
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> 20mo
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> heap
>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>> uploading
>>>>>>>>>>>>>>>>> a 500mo
>>>>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>> impossible
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> Grizzly?
>>>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>>>> notice
>>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> you're
>>>>>>>>>>>>>>>>> tied
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> Multipart
>>>>>>>>>>>>>>>>> uploads,
>>>>>>>>>>>>>>>>> you'd
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> convert
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> multipart
>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> Buffers
>>>>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FeedableBodyGenerator.
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>> a closer
>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> area
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> improved.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Btw
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>> upload.
>>>>>>>>>>>>>>>>> I receive
>>>>>>>>>>>>>>>>> a file
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> transmit
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> a storage
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> (like
>>>>>>>>>>>>>>>>> S3).
>>>>>>>>>>>>>>>>> CXF
>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>> consume
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>> bevause
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> streaming
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> large
>>>>>>>>>>>>>>>>> fle
>>>>>>>>>>>>>>>>> uploads
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>> system,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>> provides
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> input
>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>
>
>
>