users@grizzly.java.net

Re: Several weird errors under high load with grizzly in glassfish

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Thu, 27 Jan 2011 13:02:49 +0100

Hi William,

> That's true we do not use the 2.1.1 version of glassfish, only a 2.1
> with patches. We tried to switch to 2.1.1 last year but we had a
> problem with memory (it was always growing).
I remember there was an issue with some JDK Collection (if I'm not
mistaken), so we started to use different one.

> We could try with a newer version but as we only use comet in
> glassfish server, we think that it can be easier to use a standalone
> grizzly http server. This will allow us to use the lastest grizzly
> 1.x and upgrade with the lastest version easely.
Agree, if you use latest Grizzly 1.9.30 release - you'd be easily able
to integrate your app into coming Glassfish 3.1.

> According to you, is it easy to switch from glassfish to a grizzly
> standalone server ? For what I have seen, only configuration is more
> difficult.
The comet part was changed a bit between 1.0.x and 1.9.x version, but
not so much... mostly it's package name change com.sun.enterprise.* to
com.sun.grizzly.*

> Another question; during your last load test with grizzly, how many
> concurrent users can be handle by a single node when streaming is
> used (not long polling)?
Unfortunately we don't do such a tests, but Glassfish performance team
does. I can check with them and let you know.

Thanks.

WBR,
Alexey.

>
> Thanks.
>
> William.
>
> On Wed, Jan 26, 2011 at 11:16 AM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
> > wrote:
> Hi William,
>
> according to the stacktrace, you're using pretty old Glassfish 2.1,
> can you at least update it to the latest 2.1.1?
>
> Thanks.
>
>
> WBR,
> Alexey.
>
> On Jan 24, 2011, at 21:16 , Guillaume wrote:
>
>> Hi Alexey,
>>
>> Unfortunately, we are stuck with glassfish 2.1 for a long time
>> before we can change grizzly or glassfish version.
>>
>> I can't provide a test case, it is the result of a load test. I
>> don't know what can produce such "stack traces".
>>
>> We had a lot of exceptions like these during 10 minutes and it
>> stops. Sometimes, some of them appear again but not that much.
>>
>> I just wanted to know if these errors are known issues or can be
>> the result of an overload.
>>
>> If we can't find out what the problem is, I think our best chance
>> is to upgrade glassfish but without certainty. If we can't identify
>> the source of the error, then we can't be sure it won't happen again.
>>
>> If anyone has an idea, don't hesitate to contact me.
>>
>> Thanks.
>>
>> William.
>>
>> On Mon, Jan 24, 2011 at 7:41 PM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
>> > wrote:
>> Hi William,
>>
>> did you try this on Glassfish 3.0.1 or 3.1 (latest promoted build)?
>>
>> Can you pls. provide the simple testcase for it?
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>> On Jan 24, 2011, at 15:55 , william.d wrote:
>>
>>
>> Hello,
>>
>> On glassfish 2.1, I have the following errors the server.log files :
>>
>> 1./ java.lang.StackOverflowError
>>
>> Very strange and scary error but it is printed as a warning in log
>> files.
>> Does anyone have already seen this error ?
>>
>> WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|
>> _ThreadID=172;_ThreadName=httpWorkerThread-8280-466;_Reque
>> stID=355fd080-200e-4c78-b7c2-9186137e84fb;|
>> java.lang.StackOverflowError
>> at
>> org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:
>> 304)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1394)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>> (AsynchronousOutputBuffer.java:103)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>> (SocketChannelOutputBuffer.java:145)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>> (AsynchronousOutputBuffer.java:70)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 378)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:
>> 815)
>> at
>> org
>> .apache
>> .coyote
>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>> 502)
>> at
>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>> (DefaultProcessorTask.java:1457)
>> at
>> com
>> .sun
>> .enterprise
>> .web
>> .connector
>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>> at org.apache.coyote.Response.action(Response.java:235)
>> at
>> org
>> .apache
>> .coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:
>> 360)
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>> (SocketChannelOutputBuffer.java:180)
>>
>> 2./ NPE on DefaultAsyncExecutor.postExecute :
>>
>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>> _ThreadID=171;_ThreadName=httpWorkerThread-8280-86;_Requ
>> estID=73f0a3ee-5cfa-4bca-a70e-c3938863bc03;|The log message is null.
>> java.lang.NullPointerException
>> at
>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.postExecute
>> (DefaultAsyncExecutor.java:189)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>> (AsyncProcessorTask.java:99)
>> at
>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:
>> 265)
>> at
>> com
>> .sun
>> .enterprise
>> .web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:
>> 116)
>>
>>
>> 3./ NPE in CometEngine.handle
>>
>> For this one, it seems that the selection key is null (by looking
>> at the
>> code). Can it be reproduced by a use case ?
>>
>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>> _ThreadID=170;_ThreadName=httpWorkerThread-8280-43;_Requ
>> estID=1d902f71-86f0-46f7-9d66-7e9cb47f729f;|The log message is null.
>> java.lang.NullPointerException
>> at
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:
>> 881)
>> at
>> com
>> .sun
>> .enterprise
>> .web.connector.grizzly.comet.CometEngine.handle(CometEngine.java:346)
>> at
>> com.sun.enterprise.web.connector.grizzly.comet.CometAsyncFilter.doFilter
>> (CometAsyncFilter.java:84)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.invokeFilters
>> (DefaultAsyncExecutor.java:175)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.interrupt
>> (DefaultAsyncExecutor.java:153)
>> at
>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>> (AsyncProcessorTask.java:92)
>> at
>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:
>> 265)
>> at
>> com
>> .sun
>> .enterprise
>> .web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:
>> 116)
>> |#]
>>
>>
>> Errors 2. and 3. seems to be related to async processing.... Do you
>> have any
>> idea why a selection key or a processor task could be null ?
>>
>> 4. NPE in a ressource bundle.
>>
>> For the last one it seems that a bundle is missing but I don't know
>> how to
>> finf it in order to fix it.
>>
>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>> _ThreadID=120;_ThreadName=httpWorkerThread-8280-234;_Req
>> uestID=84dd1cd9-036c-43fc-930a-af79dc664a6a;|
>> StandardWrapperValve[update]:
>> Cannot find message associated with key
>> standardWrapper.acknowledgeExceptio
>> n
>> java.lang.NullPointerException
>> |#]
>>
>>
>> Thanks.
>>
>> William.
>>
>>
>> --
>> View this message in context: http://old.nabble.com/Several-weird-errors-under-high-load-with-grizzly-in-glassfish-tp30748059p30748059.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>>
>>
>
>