users@grizzly.java.net

Re: NPE's at low concurrency using AsyncHttpClient

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Mon, 30 Mar 2015 13:40:29 -0700

Hi Dan,

great! Let me know if you have more information on the issue. Just to
make sure, you see the problem when you send POST to a StaticHttpHandler?

Thanks.

WBR,
Alexey.

On 23.03.15 09:41, Daniel Feist wrote:
> Hi,
>
> Sorry, I never did get to reproduce this and file an issue, it's still
> on the TODO list, just low down given it only affects POST's to static
> resource url's, not something you'd typically do.
>
> BTW, cool to see you working on http2, i'll give you a shout when I
> find some time to take it for a test-drive.
>
> Dan
>
> On Mon, Feb 23, 2015 at 6:51 PM, Oleksiy Stashok
> <oleksiy.stashok_at_oracle.com <mailto:oleksiy.stashok_at_oracle.com>> wrote:
>
> What happens if you increase concurrency and start to send POSTs
> at high rate? Does it help to reproduce the problem?
>
> WBR,
> Alexey.
>
>
> On 23.02.15 05:06, Daniel Feist wrote:
>
> I wasn't able to reproduce this in a test case yet, even
> though I tried.
>
> It appears this issues is caused by the combination of:
>
> i) Server no reading full http body
> ii) Potentially also timing
>
> Given it only happens when posting to a tomcat (also jetty)
> static resource.
>
> Dan
>
> On Tue, Feb 17, 2015 at 4:50 AM, Oleksiy Stashok
> <oleksiy.stashok_at_oracle.com
> <mailto:oleksiy.stashok_at_oracle.com>> wrote:
>
> Hi Dan,
>
> you say it happens at low concurrency, so it shouldn't be
> difficult to
> reproduce the failure.
> Can you pls. share the testcase, it would be much easier
> for me to figure
> out what's wrong.
>
> Thank you.
>
> WBR,
> Alexey.
>
>
> On 16.02.15 03:45, Daniel Feist wrote:
>
> Hi,
>
> After investigating further this is what I found.
>
> It occured to me that the problem may not be with
> AHC/Grizzly but
> rather provoked by the back-end service I was
> hitting. So I switched
> out the back-end implementation to see what happened:
>
> Tomcat 6 hitting static resource (file in webapps/ROOT)
> --------------------------------------------------------------------------
> This is what causes the NPE's. I realise it's
> probably a bad idea to
> be testing with a POST request against a static
> resource, there was an
> error in the test url we were using, but even so I
> wouldn't expect
> NPE's client side. The http body size is irrelevant
> and this only
> occurs when InputStreamBodyGenerator is used. As i
> mentioned before
> failures rate is 2-5% of requests.
>
> Tomcat/Grizzly/Netty Echo Service
> -----------------------------------------------
> Using a echo service that reads in request stream from
> post and then
> echo's response the NPE's go away but there is still
> always a singe
> failed request.
>
> Exception stack is:
> 1. Locally closed (java.io.IOException)
> 2. java.io.IOException: Locally closed
> (java.util.concurrent.ExecutionException)
> org.glassfish.grizzly.impl.SafeFutureImpl$Sync:363
> (null)
> 3. java.util.concurrent.ExecutionException:
> java.io.IOException:
> Locally closed (java.io.IOException)
>
> org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:274
> (null)
>
> In what cases does "locally closed" IOException occur
> in grizzly? It
> seems strange that it's consistently seen exactly once
> even when we
> test with millions of request.
>
> Note: When we tested with a Jetty echo implementation
> we saw no errors
> at all, but with tomcat, grizzly, netty exactly one,
> weird!
>
> I'll keep looking into this, but any pointers would be
> great!
>
> thanks,
>
> Dan
>
>
> On Fri, Feb 13, 2015 at 6:15 PM, Daniel Feist
> <dfeist_at_gmail.com <mailto:dfeist_at_gmail.com>> wrote:
>
> Hi,
>
> 1.8.x. We were originally using 1.8.14, but
> testing with 1.8.15 (which
> has some changes to InputStreamBodyGenerator) made
> no difference.
>
> Dan
>
> On Fri, Feb 13, 2015 at 5:37 PM, Oleksiy Stashok
> <oleksiy.stashok_at_oracle.com
> <mailto:oleksiy.stashok_at_oracle.com>> wrote:
>
> Hi Dan,
>
> which AHC version are you using?
>
> WBR,
> Alexey.
>
>
> On 13.02.15 07:35, Daniel Feist wrote:
>
> Hi Alexey,
>
> We're geting a weird issue when using
> AHC/Grizzly for outbound HTTP
> POST operations. When testing using POST
> and giving AHC an
> InputStream to send, we get a significant
> number of NPE's occuring (1
> to 5% of requests), even at just 10 client
> threads. I havn't been
> able to reproduce this locally yet, but it
> happens consistently on our
> test enviroment.
>
> I'm not sure how to get to the bottom of
> this (am in the middle of
> trying) but given your much deeper
> knowledge of the codebase maybe you
> have some insights/ideas. With a single
> client thread there are no
> issues so it look like it may be some kind
> of race condition. Also
> passing a String to AHC or turn off
> chunking such that
> com.ning.http.client.generators.InputStreamBodyGenerator
> doesn't get
> used, there are no NPE's either.
>
> Dan
>
> -----
>
> This is the error seen:
>
> WARN 2015-02-13 09:06:38,336
> [[http_module_proxy].proxyConfig.worker.10]
> org.glassfish.grizzly.filterchain.DefaultFilterChain:
> GRIZZLY0013:
> Exception during FilterChain execution
> java.lang.NullPointerException
> at
>
> com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onHttpContentEncoded(GrizzlyAsyncHttpProvider.java:1243)
> ~[?:?]
> at
>
> org.glassfish.grizzly.http.HttpCodecFilter.encodeHttpPacket(HttpCodecFilter.java:1463)
> ~[?:?]
> at
>
> org.glassfish.grizzly.http.HttpClientFilter.encodeHttpPacket(HttpClientFilter.java:283)
> ~[?:?]
>
>
>
>