Hi,
Sorry, I never did get to reproduce this and file an issue, it's still on
the TODO list, just low down given it only affects POST's to static
resource url's, not something you'd typically do.
BTW, cool to see you working on http2, i'll give you a shout when I find
some time to take it for a test-drive.
Dan
On Mon, Feb 23, 2015 at 6:51 PM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
> wrote:
> What happens if you increase concurrency and start to send POSTs at high
> rate? Does it help to reproduce the problem?
>
> WBR,
> Alexey.
>
>
> On 23.02.15 05:06, Daniel Feist wrote:
>
>> I wasn't able to reproduce this in a test case yet, even though I tried.
>>
>> It appears this issues is caused by the combination of:
>>
>> i) Server no reading full http body
>> ii) Potentially also timing
>>
>> Given it only happens when posting to a tomcat (also jetty) static
>> resource.
>>
>> Dan
>>
>> On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
>> <oleksiy.stashok_at_oracle.com> wrote:
>>
>>> Hi Dan,
>>>
>>> you say it happens at low concurrency, so it shouldn't be difficult to
>>> reproduce the failure.
>>> Can you pls. share the testcase, it would be much easier for me to figure
>>> out what's wrong.
>>>
>>> Thank you.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>
>>> On 16.02.15 03:45, Daniel Feist wrote:
>>>
>>>> Hi,
>>>>
>>>> After investigating further this is what I found.
>>>>
>>>> It occured to me that the problem may not be with AHC/Grizzly but
>>>> rather provoked by the back-end service I was hitting. So I switched
>>>> out the back-end implementation to see what happened:
>>>>
>>>> Tomcat 6 hitting static resource (file in webapps/ROOT)
>>>> ------------------------------------------------------------
>>>> --------------
>>>> This is what causes the NPE's. I realise it's probably a bad idea to
>>>> be testing with a POST request against a static resource, there was an
>>>> error in the test url we were using, but even so I wouldn't expect
>>>> NPE's client side. The http body size is irrelevant and this only
>>>> occurs when InputStreamBodyGenerator is used. As i mentioned before
>>>> failures rate is 2-5% of requests.
>>>>
>>>> Tomcat/Grizzly/Netty Echo Service
>>>> -----------------------------------------------
>>>> Using a echo service that reads in request stream from post and then
>>>> echo's response the NPE's go away but there is still always a singe
>>>> failed request.
>>>>
>>>> Exception stack is:
>>>> 1. Locally closed (java.io.IOException)
>>>> 2. java.io.IOException: Locally closed
>>>> (java.util.concurrent.ExecutionException)
>>>> org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363 (null)
>>>> 3. java.util.concurrent.ExecutionException: java.io.IOException:
>>>> Locally closed (java.io.IOException)
>>>> org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
>>>> (null)
>>>>
>>>> In what cases does "locally closed" IOException occur in grizzly? It
>>>> seems strange that it's consistently seen exactly once even when we
>>>> test with millions of request.
>>>>
>>>> Note: When we tested with a Jetty echo implementation we saw no errors
>>>> at all, but with tomcat, grizzly, netty exactly one, weird!
>>>>
>>>> I'll keep looking into this, but any pointers would be great!
>>>>
>>>> thanks,
>>>>
>>>> Dan
>>>>
>>>>
>>>> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist <dfeist_at_gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> 1.8.x. We were originally using 1.8.14, but testing with 1.8.15 (which
>>>>> has some changes to InputStreamBodyGenerator) made no difference.
>>>>>
>>>>> Dan
>>>>>
>>>>> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
>>>>> <oleksiy.stashok_at_oracle.com> wrote:
>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> which AHC version are you using?
>>>>>>
>>>>>> WBR,
>>>>>> Alexey.
>>>>>>
>>>>>>
>>>>>> On 13.02.15 07:35, Daniel Feist wrote:
>>>>>>
>>>>>>> Hi Alexey,
>>>>>>>
>>>>>>> We're geting a weird issue when using AHC/Grizzly for outbound HTTP
>>>>>>> POST operations. When testing using POST and giving AHC an
>>>>>>> InputStream to send, we get a significant number of NPE's occuring (1
>>>>>>> to 5% of requests), even at just 10 client threads. I havn't been
>>>>>>> able to reproduce this locally yet, but it happens consistently on
>>>>>>> our
>>>>>>> test enviroment.
>>>>>>>
>>>>>>> I'm not sure how to get to the bottom of this (am in the middle of
>>>>>>> trying) but given your much deeper knowledge of the codebase maybe
>>>>>>> you
>>>>>>> have some insights/ideas. With a single client thread there are no
>>>>>>> issues so it look like it may be some kind of race condition. Also
>>>>>>> passing a String to AHC or turn off chunking such that
>>>>>>> com.ning.http.client.generators.InputStreamBodyGenerator doesn't get
>>>>>>> used, there are no NPE's either.
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>> -----
>>>>>>>
>>>>>>> This is the error seen:
>>>>>>>
>>>>>>> WARN 2015-02-13 09:06:38,336
>>>>>>> [[http_module_proxy].proxyConfig.worker.10]
>>>>>>> org.glassfish.grizzly.filterchain.DefaultFilterChain: GRIZZLY0013:
>>>>>>> Exception during FilterChain execution
>>>>>>> java.lang.NullPointerException
>>>>>>> at
>>>>>>>
>>>>>>> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$
>>>>>>> AsyncHttpClientEventFilter.onHttpContentEncoded(
>>>>>>> GrizzlyAsyncHttpProvider.java:1243)
>>>>>>> ~[?:?]
>>>>>>> at
>>>>>>>
>>>>>>> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(
>>>>>>> HttpCodecFilter.java:1463)
>>>>>>> ~[?:?]
>>>>>>> at
>>>>>>>
>>>>>>> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(
>>>>>>> HttpClientFilter.java:283)
>>>>>>> ~[?:?]
>>>>>>>
>>>>>>
>>>>>>
>