What happens if you increase concurrency and start to send POSTs at high
rate? Does it help to reproduce the problem?
WBR,
Alexey.
On 23.02.15 05:06, Daniel Feist wrote:
> I wasn't able to reproduce this in a test case yet, even though I tried.
>
> It appears this issues is caused by the combination of:
>
> i) Server no reading full http body
> ii) Potentially also timing
>
> Given it only happens when posting to a tomcat (also jetty) static resource.
>
> Dan
>
> On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
> <oleksiy.stashok_at_oracle.com> wrote:
>> Hi Dan,
>>
>> you say it happens at low concurrency, so it shouldn't be difficult to
>> reproduce the failure.
>> Can you pls. share the testcase, it would be much easier for me to figure
>> out what's wrong.
>>
>> Thank you.
>>
>> WBR,
>> Alexey.
>>
>>
>> On 16.02.15 03:45, Daniel Feist wrote:
>>> Hi,
>>>
>>> After investigating further this is what I found.
>>>
>>> It occured to me that the problem may not be with AHC/Grizzly but
>>> rather provoked by the back-end service I was hitting. So I switched
>>> out the back-end implementation to see what happened:
>>>
>>> Tomcat 6 hitting static resource (file in webapps/ROOT)
>>> --------------------------------------------------------------------------
>>> This is what causes the NPE's. I realise it's probably a bad idea to
>>> be testing with a POST request against a static resource, there was an
>>> error in the test url we were using, but even so I wouldn't expect
>>> NPE's client side. The http body size is irrelevant and this only
>>> occurs when InputStreamBodyGenerator is used. As i mentioned before
>>> failures rate is 2-5% of requests.
>>>
>>> Tomcat/Grizzly/Netty Echo Service
>>> -----------------------------------------------
>>> Using a echo service that reads in request stream from post and then
>>> echo's response the NPE's go away but there is still always a singe
>>> failed request.
>>>
>>> Exception stack is:
>>> 1. Locally closed (java.io.IOException)
>>> 2. java.io.IOException: Locally closed
>>> (java.util.concurrent.ExecutionException)
>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
>>> 3. java.util.concurrent.ExecutionException: java.io.IOException:
>>> Locally closed (java.io.IOException)
>>> org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
>>> (null)
>>>
>>> In what cases does "locally closed" IOException occur in grizzly? It
>>> seems strange that it's consistently seen exactly once even when we
>>> test with millions of request.
>>>
>>> Note: When we tested with a Jetty echo implementation we saw no errors
>>> at all, but with tomcat, grizzly, netty exactly one, weird!
>>>
>>> I'll keep looking into this, but any pointers would be great!
>>>
>>> thanks,
>>>
>>> Dan
>>>
>>>
>>> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <dfeist_at_gmail.com> wrote:
>>>> Hi,
>>>>
>>>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>>>> has some changes to InputStreamBodyGenerator) made no difference.
>>>>
>>>> Dan
>>>>
>>>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>>>> <oleksiy.stashok_at_oracle.com> wrote:
>>>>> Hi Dan,
>>>>>
>>>>> which AHC version are you using?
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>
>>>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>>>> Hi Alexey,
>>>>>>
>>>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>>>> POST operations. When testing using POST and giving AHC an
>>>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>>>> to 5% of requests), even at just 10 client threads. I havn't been
>>>>>> able to reproduce this locally yet, but it happens consistently on our
>>>>>> test enviroment.
>>>>>>
>>>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>>>> trying) but given your much deeper knowledge of the codebase maybe you
>>>>>> have some insights/ideas. With a single client thread there are no
>>>>>> issues so it look like it may be some kind of race condition. Also
>>>>>> passing a String to AHC or turn off chunking such that
>>>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>>>> used, there are no NPE's either.
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>> -----
>>>>>>
>>>>>> This is the error seen:
>>>>>>
>>>>>> WARN 2015-02-13 09:06:38,336
>>>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>>>> Exception during FilterChain execution
>>>>>> java.lang.NullPointerException
>>>>>> at
>>>>>>
>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>>>>> ~[?:?]
>>>>>> at
>>>>>>
>>>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>>>>> ~[?:?]
>>>>>> at
>>>>>>
>>>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>>>>> ~[?:?]
>>>>>