users@grizzly.java.net

Re: Several weird errors under high load with grizzly in glassfish

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Tue, 1 Feb 2011 10:55:54 +0100

Hi William,

sorry for the late response...

> Our goal is to use grizzly as a standalone server and stop using
> glassfish server. Glassfish provide a lot of stuff that we don't
> need so we think we only need grizzly. Do you think it is a good
> solution ?
Yes, why not. If you need HTTP part - sure.

> I have tried to follow the documentation link about embeddable
> server (Embeddable HTTP WebServer Getting Started!) on grizzly web
> page the link is broken. I found some information on JF Arcand's
> blog. Do you have a document that explain how to configure girzzly
> as a standalone web server (ie available properties, parameters,
> etc) ?
Generally there are 2 properties, which IMO are important: number of
selector threads, number of worker threads (min/max):

(1) selectorThread.setReadThreadsCount(<int>)
(2) selectorThread.setCoreThreads(<int>),
selectorThread.setMaxThreads(<int>)

Here is the reference to Scott's blog [1], where he suggests how the
numbers might be chosen (the "Tune HTTP threads" section). In that
blog "acceptor threads" are actually "selector threads", which are set
using (1), "request-processing threads" are worker threads, which are
set via (2).

Hope this will help.

WBR,
Alexey.

[1] http://weblogs.java.net/blog/2007/12/03/glassfish-tuning-primer

>
> Thanks,
>
> William.
>
> On Thu, Jan 27, 2011 at 1:02 PM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
> > wrote:
> Hi William,
>
>> That's true we do not use the 2.1.1 version of glassfish, only a
>> 2.1 with patches. We tried to switch to 2.1.1 last year but we had
>> a problem with memory (it was always growing).
> I remember there was an issue with some JDK Collection (if I'm not
> mistaken), so we started to use different one.
>
>> We could try with a newer version but as we only use comet in
>> glassfish server, we think that it can be easier to use a
>> standalone grizzly http server. This will allow us to use the
>> lastest grizzly 1.x and upgrade with the lastest version easely.
> Agree, if you use latest Grizzly 1.9.30 release - you'd be easily
> able to integrate your app into coming Glassfish 3.1.
>
>> According to you, is it easy to switch from glassfish to a grizzly
>> standalone server ? For what I have seen, only configuration is
>> more difficult.
> The comet part was changed a bit between 1.0.x and 1.9.x version,
> but not so much... mostly it's package name change
> com.sun.enterprise.* to com.sun.grizzly.*
>
>> Another question; during your last load test with grizzly, how many
>> concurrent users can be handle by a single node when streaming is
>> used (not long polling)?
> Unfortunately we don't do such a tests, but Glassfish performance
> team does. I can check with them and let you know.
>
> Thanks.
>
> WBR,
> Alexey.
>
>>
>> Thanks.
>>
>> William.
>>
>> On Wed, Jan 26, 2011 at 11:16 AM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
>> > wrote:
>> Hi William,
>>
>> according to the stacktrace, you're using pretty old Glassfish 2.1,
>> can you at least update it to the latest 2.1.1?
>>
>> Thanks.
>>
>>
>> WBR,
>> Alexey.
>>
>> On Jan 24, 2011, at 21:16 , Guillaume wrote:
>>
>>> Hi Alexey,
>>>
>>> Unfortunately, we are stuck with glassfish 2.1 for a long time
>>> before we can change grizzly or glassfish version.
>>>
>>> I can't provide a test case, it is the result of a load test. I
>>> don't know what can produce such "stack traces".
>>>
>>> We had a lot of exceptions like these during 10 minutes and it
>>> stops. Sometimes, some of them appear again but not that much.
>>>
>>> I just wanted to know if these errors are known issues or can be
>>> the result of an overload.
>>>
>>> If we can't find out what the problem is, I think our best chance
>>> is to upgrade glassfish but without certainty. If we can't
>>> identify the source of the error, then we can't be sure it won't
>>> happen again.
>>>
>>> If anyone has an idea, don't hesitate to contact me.
>>>
>>> Thanks.
>>>
>>> William.
>>>
>>> On Mon, Jan 24, 2011 at 7:41 PM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
>>> > wrote:
>>> Hi William,
>>>
>>> did you try this on Glassfish 3.0.1 or 3.1 (latest promoted build)?
>>>
>>> Can you pls. provide the simple testcase for it?
>>>
>>> Thanks.
>>>
>>> WBR,
>>> Alexey.
>>>
>>> On Jan 24, 2011, at 15:55 , william.d wrote:
>>>
>>>
>>> Hello,
>>>
>>> On glassfish 2.1, I have the following errors the server.log files :
>>>
>>> 1./ java.lang.StackOverflowError
>>>
>>> Very strange and scary error but it is printed as a warning in log
>>> files.
>>> Does anyone have already seen this error ?
>>>
>>> WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|
>>> _ThreadID=172;_ThreadName=httpWorkerThread-8280-466;_Reque
>>> stID=355fd080-200e-4c78-b7c2-9186137e84fb;|
>>> java.lang.StackOverflowError
>>> at
>>> org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:
>>> 304)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1394)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:
>>> 502)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web
>>> .connector
>>> .grizzly.DefaultProcessorTask.action(DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>> org
>>> .apache
>>> .coyote
>>> .http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>>
>>> 2./ NPE on DefaultAsyncExecutor.postExecute :
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>>> _ThreadID=171;_ThreadName=httpWorkerThread-8280-86;_Requ
>>> estID=73f0a3ee-5cfa-4bca-a70e-c3938863bc03;|The log message is null.
>>> java.lang.NullPointerException
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.postExecute
>>> (DefaultAsyncExecutor.java:189)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>>> (AsyncProcessorTask.java:99)
>>> at
>>> com
>>> .sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:
>>> 265)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:
>>> 116)
>>>
>>>
>>> 3./ NPE in CometEngine.handle
>>>
>>> For this one, it seems that the selection key is null (by looking
>>> at the
>>> code). Can it be reproduced by a use case ?
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>>> _ThreadID=170;_ThreadName=httpWorkerThread-8280-43;_Requ
>>> estID=1d902f71-86f0-46f7-9d66-7e9cb47f729f;|The log message is null.
>>> java.lang.NullPointerException
>>> at
>>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:
>>> 881)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web.connector.grizzly.comet.CometEngine.handle(CometEngine.java:
>>> 346)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.comet.CometAsyncFilter.doFilter
>>> (CometAsyncFilter.java:84)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.invokeFilters
>>> (DefaultAsyncExecutor.java:175)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.interrupt
>>> (DefaultAsyncExecutor.java:153)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>>> (AsyncProcessorTask.java:92)
>>> at
>>> com
>>> .sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:
>>> 265)
>>> at
>>> com
>>> .sun
>>> .enterprise
>>> .web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:
>>> 116)
>>> |#]
>>>
>>>
>>> Errors 2. and 3. seems to be related to async processing.... Do
>>> you have any
>>> idea why a selection key or a processor task could be null ?
>>>
>>> 4. NPE in a ressource bundle.
>>>
>>> For the last one it seems that a bundle is missing but I don't
>>> know how to
>>> finf it in order to fix it.
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|
>>> _ThreadID=120;_ThreadName=httpWorkerThread-8280-234;_Req
>>> uestID=84dd1cd9-036c-43fc-930a-af79dc664a6a;|
>>> StandardWrapperValve[update]:
>>> Cannot find message associated with key
>>> standardWrapper.acknowledgeExceptio
>>> n
>>> java.lang.NullPointerException
>>> |#]
>>>
>>>
>>> Thanks.
>>>
>>> William.
>>>
>>>
>>> --
>>> View this message in context: http://old.nabble.com/Several-weird-errors-under-high-load-with-grizzly-in-glassfish-tp30748059p30748059.html
>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>
>>>
>>>
>>>
>>
>>
>
>