Hi,
After investigating further this is what I found.
It occured to me that the problem may not be with AHC/Grizzly but
rather provoked by the back-end service I was hitting. So I switched
out the back-end implementation to see what happened:
Tomcat 6 hitting static resource (file in webapps/ROOT)
--------------------------------------------------------------------------
This is what causes the NPE's. I realise it's probably a bad idea to
be testing with a POST request against a static resource, there was an
error in the test url we were using, but even so I wouldn't expect
NPE's client side. The http body size is irrelevant and this only
occurs when InputStreamBodyGenerator is used. As i mentioned before
failures rate is 2-5% of requests.
Tomcat/Grizzly/Netty Echo Service
-----------------------------------------------
Using a echo service that reads in request stream from post and then
echo's response the NPE's go away but there is still always a singe
failed request.
Exception stack is:
1. Locally closed (java.io.IOException)
2. java.io.IOException: Locally closed (java.util.concurrent.ExecutionException)
org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
3. java.util.concurrent.ExecutionException: java.io.IOException:
Locally closed (java.io.IOException)
org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274 (null)
In what cases does "locally closed" IOException occur in grizzly? It
seems strange that it's consistently seen exactly once even when we
test with millions of request.
Note: When we tested with a Jetty echo implementation we saw no errors
at all, but with tomcat, grizzly, netty exactly one, weird!
I'll keep looking into this, but any pointers would be great!
thanks,
Dan
On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <dfeist_at_gmail.com> wrote:
> Hi,
>
> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
> has some changes to InputStreamBodyGenerator) made no difference.
>
> Dan
>
> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
> <oleksiy.stashok_at_oracle.com> wrote:
>> Hi Dan,
>>
>> which AHC version are you using?
>>
>> WBR,
>> Alexey.
>>
>>
>> On 13.02.15 07:35, Daniel Feist wrote:
>>>
>>> Hi Alexey,
>>>
>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>> POST operations. When testing using POST and giving AHC an
>>> InputStream to send, we get a significant number of NPE's occuring (1
>>> to 5% of requests), even at just 10 client threads. I havn't been
>>> able to reproduce this locally yet, but it happens consistently on our
>>> test enviroment.
>>>
>>> I'm not sure how to get to the bottom of this (am in the middle of
>>> trying) but given your much deeper knowledge of the codebase maybe you
>>> have some insights/ideas. With a single client thread there are no
>>> issues so it look like it may be some kind of race condition. Also
>>> passing a String to AHC or turn off chunking such that
>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>> used, there are no NPE's either.
>>>
>>> Dan
>>>
>>> -----
>>>
>>> This is the error seen:
>>>
>>> WARN 2015-02-13 09:06:38,336
>>> [[http_module_proxy].proxyConfig.worker.10]
>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>> Exception during FilterChain execution
>>> java.lang.NullPointerException
>>> at
>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
>>> ~[?:?]
>>> at
>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
>>> ~[?:?]
>>> at
>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
>>> ~[?:?]
>>
>>