What is very strange is that I tested with/without the same sessionCode
with our previous code, the one not using FeedableBodyGenerator, which has
a high memory consumption.
Despites the fact it had high memory consumption, it seems work fine to
upload multiple documents if allocated with a large heap, and the
sessionCode seems to have no effect.
On the new impl using the FeedableBodyGenerator, the sessionCode sent as a
multipart bodypart seems to have an effect.
I have tried to feed the queue before sending the request to AHC, but this
leads to this exception (with/without sessionCode switching)
Caused by: java.util.concurrent.TimeoutException: Timeout exceeded
at
com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:528)
at
com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:361)
at
org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:383)
at
org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:362)
at
org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
> By the way, by using a low timeout with the same sessioncode, I got the
> following NPE:
>
> Caused by: java.lang.NullPointerException
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:184)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
> at
> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>
> GrizzlyAsyncHttpProvider.HttpTransactionContext httpCtx =
> getHttpTransactionContext(c);
> httpCtx.abort(e.getCause());
>
> I guess the httpCtx is not already available to be aborted
>
>
> 2013/9/5 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>
>>
>>
>> This is right, here's a log I have when I use the same session code, ie
>> the remote host is blocking the data or something.
>> This is obtained by running 5 parallel uploads.
>>
>> *Flushing queue of size 0 with allowBlocking = false*
>> Flushing queue of size 1 with allowBlocking = true
>> *Flushing queue of size 97 with allowBlocking = false*
>> *Flushing queue of size 100 with allowBlocking = false*
>> *Flushing queue of size 160 with allowBlocking = true*
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 0 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> Dumping heap to /ome/lorber/ureau/om ...
>> Unable to create /ome/lorber/ureau/om: Le fichier existe
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Disconnected from the target VM, address: '127.0.0.1:49268', transport:
>> 'socket'
>>
>>
>>
>>
>>
>> Otherwise, with different session codes, I get the following:
>>
>> *Flushing queue of size 0 with allowBlocking = false*
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> *Flushing queue of size 0 with allowBlocking = false*
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> *Flushing queue of size 0 with allowBlocking = false*
>> *Flushing queue of size 0 with allowBlocking = false*
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> Flushing queue of size 1 with allowBlocking = true
>> ... and this continues without OOM
>>
>>
>> So, this seems to be the problem.
>>
>>
>>
>> I think it would be great to be able to be able to choose the queue impl
>> behind that FeedableBodyGenerator, like I suggested in my pull request.
>>
>> See here:
>>
>> https://github.com/slorber/async-http-client/blob/79b0c3b28a61b0aa4c4b055bca8f6be11d9ed1e6/src/main/java/com/ning/http/client/providers/grizzly/FeedableBodyGenerator.java
>>
>> Using a LinkedBlockingQueue seems to be a nice idea in this context, and
>> in my case I would probably use a queue of size 1
>>
>>
>> This would handle the blocking of the feed method, without having to use
>> this:
>> if (context != null) {
>> flushQueue(true);
>> }
>>
>>
>> Or perhaps the feed() method have to wait until a context is set in the
>> BodyGenerator ?
>>
>>
>> I think it would be more clear if the initializeAsynchronousTransfer
>> simply didn't flush the queue but just setup the context.
>> Then the feed method would block until there's a context set, and then
>> flush the queue with blocking behavior.
>>
>> This is probably the next step, but as we are using AHC for async, it
>> would probably be great if that blocking feed() method was called in a
>> worker thread instead of our main thread.
>> I won't use this but someone who really wants a non-blocking impl of
>> performant multipart fileupload would probably need this, or will use an
>> ExecutorService for the feeding operations as a workaround.
>>
>>
>>
>> Thanks again for your reactivity
>>
>>
>>
>>
>>
>> 2013/9/4 Ryan Lubke <ryan.lubke_at_oracle.com>
>>
>>>
>>>
>>> Sébastien Lorber wrote:
>>>
>>> I've integrated this change and it works fine except a little detail.
>>>
>>>
>>>
>>>
>>> I'm uploading files to a third party API (a bit like S3).
>>> This API requires a "sessionCode" in each request. So there is a
>>> multipart StringPart with that SessionCode.
>>>
>>> We used to have a cache which holds the sessionCode 30min per user so
>>> that we do not need to init a new session each time.
>>>
>>> I had troubles in this very specific case: when I upload 5 docs with the
>>> same session code.
>>> When I remove the cache and use 5 different session codes, it works fine.
>>>
>>> I guess the remote service is blocking concurrent uploads with the same
>>> session code. I don't know at all.
>>>
>>>
>>>
>>> Where I want to go is that I wouldn't have expected Grizzly to OOM
>>>
>>>
>>> Avertissement: Exception during FilterChain execution
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>> at
>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>> at
>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:238)
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:405)
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:320)
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:263)
>>> at org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2500)
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext$1.run(FilterChainContext.java:196)
>>> at
>>> org.glassfish.grizzly.filterchain.FilterChainContext.resume(FilterChainContext.java:220)
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter$SSLHandshakeContext.completed(SSLFilter.java:383)
>>> at
>>> org.glassfish.grizzly.ssl.SSLFilter.notifyHandshakeComplete(SSLFilter.java:278)
>>> at
>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:275)
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2490)
>>> at
>>> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>> at
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>> at
>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>> at
>>> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:546)
>>> at
>>> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
>>>
>>>
>>>
>>>
>>>
>>> Caused by: java.util.concurrent.TimeoutException: null
>>> at
>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:367)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:274)
>>> ~[grizzly-framework-2.3.5.jar:2.3.5]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.block(FeedableBodyGenerator.java:177)
>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.blockUntilQueueFree(FeedableBodyGenerator.java:167)
>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:124)
>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>> at
>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:94)
>>> ~[async-http-client-1.7.20-204092c.jar:na]
>>>
>>>
>>>
>>> multipart.body.generator.feeder.buffer=100000 -> size of each Buffer
>>> sent to the FeedableBodyGenerator
>>> transport.max.pending.bytes=1000000
>>> (I tried other settings, including AUTO_SIZE)
>>>
>>>
>>>
>>> Do you have any idea why is there an OOM with these settings?
>>>
>>>
>>>
>>>
>>> Perhaps it is because the feed() method of FeedableBodyGenerator doesn't
>>> block until the context is initialized.
>>> I guess the initializeAsynchronousTransfer is called only once the
>>> connection is established, and perhaps a lot of Buffer are added to the
>>> queue...
>>>
>>> Yes, it's invoked once the request has been dispatched, so if the
>>> generator is fed a lot before the request, I could see this happening.
>>> I'll see what I can do to alleviate that case.
>>>
>>>
>>> But I'm not sure at all because the session code is transmitted as a
>>> BodyPart and I get the same problem if i put it as the first or last
>>> multipart.
>>>
>>> It's not a big deal, perhaps I should always use a different session
>>> code for concurrent operations but I'd like to be sure that we won't have
>>> this issue in production...
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/9/3 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>
>>>> Good catch. Fixed.
>>>>
>>>>
>>>> Sébastien Lorber wrote:
>>>>
>>>>
>>>>
>>>> Hello,
>>>>
>>>>
>>>> There's a little mistake in the grizzly ahc provider relative to the
>>>> write queue size.
>>>>
>>>>
>>>>
>>>> https://github.com/AsyncHttpClient/async-http-client/blob/b5d97efe9fe14113ea92fb1f7db192a2d090fad7/src/main/java/com/ning/http/client/providers/grizzly/GrizzlyAsyncHttpProvider.java#L419
>>>>
>>>> As you can see, the TransportCustomizer is called, and then the write
>>>> queue size (among other things) is set to AUTO_SIZE (instead of previously
>>>> UNLIMITED)
>>>>
>>>>
>>>>
>>>>
>>>> clientTransport.getAsyncQueueIO().getWriter()
>>>>
>>>>
>>>>
>>>> .setMaxPendingBytesPerConnection(AsyncQueueWriter.AUTO_SIZE);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I think the default settings like this AUTO_SIZE attribute should be
>>>> set before the customization of the transport, or they would override the
>>>> value we customized.
>>>>
>>>>
>>>> This is actually my case, since I can't reproduce my "bug" which is
>>>> "high memory consumption", even when using -1 / UNLIMITED in the
>>>> TransportCustomizer.
>>>>
>>>>
>>>> This could work fine for me with AUTO_SIZE, but I'd rather be able to
>>>> tune this parameter during load tests to see the effect.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/8/31 Sebastien Lorber <lorber.sebastien_at_gmail.com>
>>>>
>>>>> Thanks i will ckeck that on monday. I can now upload a 500m file with
>>>>> 40m heap size ;)
>>>>>
>>>>> Envoyé de mon iPhone
>>>>>
>>>>> Le 30 août 2013 à 20:49, Ryan Lubke <ryan.lubke_at_oracle.com> a écrit :
>>>>>
>>>>> I'm going to be updating the Grizzly provider such that AUTO_SIZE (not
>>>>> AUTO_TUNE) is the default, so you can avoid the use of the
>>>>> TransportCustomizer.
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>
>>>>> Regarding your tuning question, I would probably set the value to
>>>>> AsyncQueueWriter.AUTO_TUNE (this will be four times the socket write
>>>>> buffer) and see how that works.
>>>>>
>>>>> Ryan Lubke wrote:
>>>>>
>>>>> A question first. With these changes, your memory usage is more
>>>>> inline with what you were looking for?
>>>>>
>>>>> Sébastien Lorber wrote:
>>>>>
>>>>> By the way, do you have any idea when the 1.7.20 will be released
>>>>> (with these new improvements?)
>>>>>
>>>>> We would like to know if we wait for a release or if we install our
>>>>> own temp release on Nexus :)
>>>>>
>>>>>
>>>>> 2013/8/30 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>
>>>>>> Thank you, it works fine!
>>>>>>
>>>>>>
>>>>>> I just had to modify a single line after your commit.
>>>>>>
>>>>>>
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider#initializeTransport
>>>>>> ->
>>>>>> clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(10000);
>>>>>>
>>>>>>
>>>>>> If I let the initial value (-1) it won't block, canWrite always
>>>>>> returns true
>>>>>>
>>>>>>
>>>>>> Btw, on AHC I didn't find any way to pass this value as a config
>>>>>> attribute, neither the size of the write buffer you talked about.
>>>>>>
>>>>>> So in the end, is there a way with current AHC code to use this
>>>>>> "canWrite = false" behavior?
>>>>>> If not, can you please provide a way to set this behavior on v1.7.20
>>>>>> ? thanks
>>>>>>
>>>>>>
>>>>>> PS: does it make sens to use the same number of bytes un the
>>>>>> feed(Buffer) method and in the setMaxPendingBytesPerConnection(10000);
>>>>>> ? do you have any tuning recommandation?
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/8/29 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>
>>>>>>> Please disregard.
>>>>>>>
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>
>>>>>>> Sébastien,
>>>>>>>
>>>>>>> Could you also provide a sample of how you're performing your feed?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -rl
>>>>>>>
>>>>>>> Ryan Lubke wrote:
>>>>>>>
>>>>>>> Sébastien,
>>>>>>>
>>>>>>> I'd recommend looking at Connection.canWrite() [1] and
>>>>>>> Connection.notifyCanWrite(WriteListener) [1]
>>>>>>>
>>>>>>> By default, Grizzly will configure the async write queue length to
>>>>>>> be four times the write buffer size (which is based off the socket write
>>>>>>> buffer).
>>>>>>> When this queue exceeds this value, canWrite() will return false.
>>>>>>>
>>>>>>> When this occurs, you can register a WriteListener to be notified
>>>>>>> when the queue length is below the configured max and then simulate blocking
>>>>>>> until the onWritePossible() callback has been invoked.
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>>
>>>>>>> final FutureImpl<Boolean> future =
>>>>>>> Futures.createSafeFuture();
>>>>>>>
>>>>>>> // Connection may be obtained by calling
>>>>>>> FilterChainContext.getConnection().
>>>>>>> connection.notifyCanWrite(new WriteHandler() {
>>>>>>>
>>>>>>> @Override
>>>>>>> public void onWritePossible() throws Exception {
>>>>>>> future.result(Boolean.TRUE);
>>>>>>> }
>>>>>>>
>>>>>>> @Override
>>>>>>> public void onError(Throwable t) {
>>>>>>> future.failure(Exceptions.makeIOException(t));
>>>>>>> }
>>>>>>> });
>>>>>>>
>>>>>>> try {
>>>>>>> final long writeTimeout = 30;
>>>>>>> future.get(writeTimeout, TimeUnit.MILLISECONDS);
>>>>>>> } catch (ExecutionException e) {
>>>>>>> HttpTransactionContext httpCtx =
>>>>>>> HttpTransactionContext.get(connection);
>>>>>>> httpCtx.abort(e.getCause());
>>>>>>> } catch (Exception e) {
>>>>>>> HttpTransactionContext httpCtx =
>>>>>>> HttpTransactionContext.get(connection);
>>>>>>> httpCtx.abort(e);
>>>>>>> }
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> [1]
>>>>>>> http://grizzly.java.net/docs/2.3/apidocs/org/glassfish/grizzly/OutputSink.html
>>>>>>> .
>>>>>>>
>>>>>>> Sébastien Lorber wrote:
>>>>>>>
>>>>>>> Ryan, I've did some other tests.
>>>>>>>
>>>>>>>
>>>>>>> It seems that using a blocking queue in the FeedableBodyGenerator is
>>>>>>> totally useless because the thread consuming it is not blocking and the
>>>>>>> queue never blocks the feeding, which was my intention in the beginning.
>>>>>>> Maybe it depends on the IO strategy used?
>>>>>>> I use AHC default which seems to use SameThreadIOStrategy so I don't
>>>>>>> think it's related to the IO strategy.
>>>>>>>
>>>>>>>
>>>>>>> So in the end I can upload a 70m file with a heap of 50m, but I have
>>>>>>> to put a Thread.sleep(30) between each 100k Buffer send to the
>>>>>>> FeedableBodyGenerator
>>>>>>>
>>>>>>> The connection with the server is not good here, but in production
>>>>>>> it is normally a lot better as far as I know.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I've tried things
>>>>>>> like clientTransport.getAsyncQueueIO().getWriter().setMaxPendingBytesPerConnection(100000);
>>>>>>> but it doesn't seem to work for me.
>>>>>>>
>>>>>>> I'd like the Grizzly internals to block when there are too much
>>>>>>> pending bytes to send. Is it possible?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> PS: I've just been able to send a 500mo file with 100mo heap, but it
>>>>>>> needed a sleep of 100ms between each 100k Buffer sent to the bodygenerator
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> By chance do you if I can remove the MessageCloner used in the SSL
>>>>>>>> filter?
>>>>>>>> SSLBaseFilter$OnWriteCopyCloner
>>>>>>>>
>>>>>>>> It seems to allocate a lot of memory.
>>>>>>>> I don't really understand why messages have to be cloned, can I
>>>>>>>> remove this? How?
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/8/29 Sébastien Lorber <lorber.sebastien_at_gmail.com>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm trying to send a 500m file for my tests with a heap of 400m.
>>>>>>>>>
>>>>>>>>> In our real use cases we would probably have files under 20mo but
>>>>>>>>> we want to reduce the memory consumption because we can have x parallel
>>>>>>>>> uploads on the same server according to the user activity.
>>>>>>>>>
>>>>>>>>> I'll try to check if using this BodyGenerator reduced the memory
>>>>>>>>> footprint or if it's almost like before.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>
>>>>>>>>>> At this point in time, as far as the SSL buffer allocation is
>>>>>>>>>> concerned, it's untunable.
>>>>>>>>>>
>>>>>>>>>> That said, feel free to open a feature request.
>>>>>>>>>>
>>>>>>>>>> As to your second question, there is no suggested size. This is
>>>>>>>>>> all very application specific.
>>>>>>>>>>
>>>>>>>>>> I'm curious, how large of a file are you sending?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>
>>>>>>>>>> I have seen a lot of buffers which have a size of 33842 and it
>>>>>>>>>> seems the limit is near half the capacity.
>>>>>>>>>>
>>>>>>>>>> Perhaps there's a way to tune that buffer size so that it
>>>>>>>>>> consumes less memory?
>>>>>>>>>> Is there an ideal Buffer size to send to the feed method?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/8/28 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>
>>>>>>>>>>> I'll be reviewing the PR today, thanks again!
>>>>>>>>>>>
>>>>>>>>>>> Regarding the OOM: as it stands now, for each new buffer that is
>>>>>>>>>>> passed to the SSLFilter, we allocate a buffer twice the size in order to
>>>>>>>>>>> accommodate the encrypted result. So there's an increase.
>>>>>>>>>>>
>>>>>>>>>>> Depending on the socket configurations of both endpoints, and
>>>>>>>>>>> how fast the remote is reading data, it could
>>>>>>>>>>> be the write queue is becoming too large. We do have a way to
>>>>>>>>>>> detect this situation, but I'm pretty sure
>>>>>>>>>>> the Grizzly internals are currently shielded here. I will see
>>>>>>>>>>> what I can do to allow users to leverage this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I've made my pull request.
>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/pull/367
>>>>>>>>>>>
>>>>>>>>>>> With my usecase it works, the file is uploaded like before.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But I didn't notice a big memory improvement.
>>>>>>>>>>>
>>>>>>>>>>> Is it possible that SSL doesn't allow to stream the body or
>>>>>>>>>>> something like that?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In memory, I have a lot of:
>>>>>>>>>>> - HeapByteBuffer
>>>>>>>>>>> Which are hold by SSLUtils$3
>>>>>>>>>>> Which are hold by BufferBuffers
>>>>>>>>>>> Which are hold by WriteResult
>>>>>>>>>>> Which are hold by AsyncWriteQueueRecord
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Here is an exemple of the OOM stacktrace:
>>>>>>>>>>>
>>>>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>>>> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
>>>>>>>>>>> at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLUtils.allocateOutputBuffer(SSLUtils.java:342)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter$2.grow(SSLBaseFilter.java:117)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.ensureBufferSize(SSLConnectionContext.java:392)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:272)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLConnectionContext.wrapAll(SSLConnectionContext.java:227)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.wrapAll(SSLBaseFilter.java:404)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLBaseFilter.handleWrite(SSLBaseFilter.java:319)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.accurateWrite(SSLFilter.java:255)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ssl.SSLFilter.handleWrite(SSLFilter.java:143)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleWrite(GrizzlyAsyncHttpProvider.java:2503)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:853)
>>>>>>>>>>> at
>>>>>>>>>>> org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:720)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.flushQueue(FeedableBodyGenerator.java:132)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.FeedableBodyGenerator.feed(FeedableBodyGenerator.java:101)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder$FeedBodyGeneratorOutputStream.write(MultipartBodyGeneratorFeeder.java:222)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>>>>>>>>>> at com.ning.http.multipart.FilePart.sendData(FilePart.java:179)
>>>>>>>>>>> at com.ning.http.multipart.Part.send(Part.java:331)
>>>>>>>>>>> at com.ning.http.multipart.Part.sendParts(Part.java:397)
>>>>>>>>>>> at
>>>>>>>>>>> com.ning.http.client.providers.grizzly.MultipartBodyGeneratorFeeder.feed(MultipartBodyGeneratorFeeder.java:144)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Any idea?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>
>>>>>>>>>>>> Excellent! Looking forward to the pull request!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan thanks, it works fine, I'll make a pull request on AHC
>>>>>>>>>>>> tomorrow with a better code using the same Part classes that already exist.
>>>>>>>>>>>>
>>>>>>>>>>>> I created an OutputStream that redirects to the BodyGenerator
>>>>>>>>>>>> feeder.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem I currently have is that the feeder feeds the queue
>>>>>>>>>>>> faster than the async thread polling it :)
>>>>>>>>>>>> I need to expose a limit to that queue size or something, will
>>>>>>>>>>>> work on that, it will be better than a thread sleep to slow down the
>>>>>>>>>>>> filepart reading
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/8/27 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, something like that. I was going to tackle adding
>>>>>>>>>>>>> something like this today. I'll follow up with something you can test out.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I see what I could do, probably something like that:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> FeedableBodyGenerator bodyGenerator = new
>>>>>>>>>>>>> FeedableBodyGenerator();
>>>>>>>>>>>>> MultipartBodyGeneratorFeeder bodyGeneratorFeeder = new
>>>>>>>>>>>>> MultipartBodyGeneratorFeeder(bodyGenerator);
>>>>>>>>>>>>> Request uploadRequest1 = new RequestBuilder("POST")
>>>>>>>>>>>>> .setUrl("url")
>>>>>>>>>>>>> .setBody(bodyGenerator)
>>>>>>>>>>>>> .build();
>>>>>>>>>>>>>
>>>>>>>>>>>>> ListenableFuture<Response> asyncRes = asyncHttpClient
>>>>>>>>>>>>> .prepareRequest(uploadRequest1)
>>>>>>>>>>>>> .execute(new AsyncCompletionHandlerBase());
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> bodyGeneratorFeeder.append("param1","value1");
>>>>>>>>>>>>> bodyGeneratorFeeder.append("param2","value2");
>>>>>>>>>>>>> bodyGeneratorFeeder.append("fileToUpload",fileInputStream);
>>>>>>>>>>>>> bodyGeneratorFeeder.end();
>>>>>>>>>>>>>
>>>>>>>>>>>>> Response uploadResponse = asyncRes.get();
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does it seem ok to you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I guess it could be interesting to provide that
>>>>>>>>>>>>> MultipartBodyGeneratorFeeder class to AHC or Grizzly since some other
>>>>>>>>>>>>> people may want to achieve the same thing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/8/26 Ryan Lubke <ryan.lubke_at_oracle.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sébastien Lorber wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would like to know if it's possible to upload a file with
>>>>>>>>>>>>>>> AHC / Grizzly in streaming, I mean without loading the whole file bytes in
>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The default behavior seems to allocate a byte[] which
>>>>>>>>>>>>>>> contans the whole file, so it means that my server can be OOM if too many
>>>>>>>>>>>>>>> users upload a large file in the same time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've tryied with a Heap and ByteBuffer memory managers, with
>>>>>>>>>>>>>>> reallocate=true/false but no more success.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems the whole file content is appended wto the
>>>>>>>>>>>>>>> BufferOutputStream, and then the underlying buffer is written.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At least this seems to be the case with AHC integration:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/AsyncHttpClient/async-http-client/blob/6faf1f316e5546110b0779a5a42fd9d03ba6bc15/providers/grizzly/src/main/java/org/asynchttpclient/providers/grizzly/bodyhandler/PartsBodyHandler.java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, is there a way to patch AHC to stream the file so that I
>>>>>>>>>>>>>>> could eventually consume only 20mo of heap while uploading a 500mo file?
>>>>>>>>>>>>>>> Or is this simply impossible with Grizzly?
>>>>>>>>>>>>>>> I didn't notice anything related to that in the
>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's possible with the FeedableBodyGenerator. But if you're
>>>>>>>>>>>>>> tied to using Multipart uploads, you'd have to convert the multipart data
>>>>>>>>>>>>>> to Buffers manually and send using the FeedableBodyGenerator.
>>>>>>>>>>>>>> I'll take a closer look to see if this area can be improved.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Btw in my case it is a file upload. I receive a file with
>>>>>>>>>>>>>>> CXF and have to transmit it to a storage server (like S3). CXF doesn't
>>>>>>>>>>>>>>> consume memory bevause it is streaming the large fle uploads to the file
>>>>>>>>>>>>>>> system, and then provides an input stream on that file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>