I wasn't able to reproduce this in a test case yet, even though I tried.
It appears this issues is caused by the combination of:
i) Server no reading full http body
ii) Potentially also timing
Given it only happens when posting to a tomcat (also jetty) static resource.
Dan
On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
<oleksiy.stashok_at_oracle.com> wrote:
> Hi Dan,
>
> you say it happens at low concurrency, so it shouldn't be difficult to
> reproduce the failure.
> Can you pls. share the testcase, it would be much easier for me to figure
> out what's wrong.
>
> Thank you.
>
> WBR,
> Alexey.
>
>
> On 16.02.15 03:45, Daniel Feist wrote:
>>
>> Hi,
>>
>> After investigating further this is what I found.
>>
>> It occured to me that the problem may not be with AHC/Grizzly but
>> rather provoked by the back-end service I was hitting. So I switched
>> out the back-end implementation to see what happened:
>>
>> Tomcat 6 hitting static resource (file in webapps/ROOT)
>> --------------------------------------------------------------------------
>> This is what causes the NPE's. I realise it's probably a bad idea to
>> be testing with a POST request against a static resource, there was an
>> error in the test url we were using, but even so I wouldn't expect
>> NPE's client side. The http body size is irrelevant and this only
>> occurs when InputStreamBodyGenerator is used. As i mentioned before
>> failures rate is 2-5% of requests.
>>
>> Tomcat/Grizzly/Netty Echo Service
>> -----------------------------------------------
>> Using a echo service that reads in request stream from post and then
>> echo's response the NPE's go away but there is still always a singe
>> failed request.
>>
>> Exception stack is:
>> 1. Locally closed (java.io.IOException)
>> 2. java.io.IOException: Locally closed
>> (java.util.concurrent.ExecutionException)
>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
>> 3. java.util.concurrent.ExecutionException: java.io.IOException:
>> Locally closed (java.io.IOException)
>> org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
>> (null)
>>
>> In what cases does "locally closed" IOException occur in grizzly? It
>> seems strange that it's consistently seen exactly once even when we
>> test with millions of request.
>>
>> Note: When we tested with a Jetty echo implementation we saw no errors
>> at all, but with tomcat, grizzly, netty exactly one, weird!
>>
>> I'll keep looking into this, but any pointers would be great!
>>
>> thanks,
>>
>> Dan
>>
>>
>> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <dfeist_at_gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>>> has some changes to InputStreamBodyGenerator) made no difference.
>>>
>>> Dan
>>>
>>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>>> <oleksiy.stashok_at_oracle.com> wrote:
>>>>
>>>> Hi Dan,
>>>>
>>>> which AHC version are you using?
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>
>>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>>>
>>>>> Hi Alexey,
>>>>>
>>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>>> POST operations. When testing using POST and giving AHC an
>>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>>> to 5% of requests), even at just 10 client threads. I havn't been
>>>>> able to reproduce this locally yet, but it happens consistently on our
>>>>> test enviroment.
>>>>>
>>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>>> trying) but given your much deeper knowledge of the codebase maybe you
>>>>> have some insights/ideas. With a single client thread there are no
>>>>> issues so it look like it may be some kind of race condition. Also
>>>>> passing a String to AHC or turn off chunking such that
>>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>>> used, there are no NPE's either.
>>>>>
>>>>> Dan
>>>>>
>>>>> -----
>>>>>
>>>>> This is the error seen:
>>>>>
>>>>> WARN 2015-02-13 09:06:38,336
>>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>>> Exception during FilterChain execution
>>>>> java.lang.NullPointerException
>>>>> at
>>>>>
>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>>>> ~[?:?]
>>>>> at
>>>>>
>>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>>>> ~[?:?]
>>>>> at
>>>>>
>>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>>>> ~[?:?]
>>>>
>>>>
>