users@grizzly.java.net

Re: Several weird errors under high load with grizzly in glassfish

From: Guillaume <itchy75_at_hotmail.fr>
Date: Wed, 26 Jan 2011 20:18:49 +0100

That's true we do not use the 2.1.1 version of glassfish, only a 2.1 with
patches. We tried to switch to 2.1.1 last year but we had a problem with
memory (it was always growing).

We could try with a newer version but as we only use comet in glassfish
server, we think that it can be easier to use a standalone grizzly http
server. This will allow us to use the lastest grizzly 1.x and upgrade with
the lastest version easely.

According to you, is it easy to switch from glassfish to a grizzly
standalone server ? For what I have seen, only configuration is more
difficult.

Another question; during your last load test with grizzly, how many
concurrent users can be handle by a single node when streaming is used (not
long polling) ?

Thanks.

William.

On Wed, Jan 26, 2011 at 11:16 AM, Oleksiy Stashok <
oleksiy.stashok_at_oracle.com> wrote:

> Hi William,
>
> according to the stacktrace, you're using pretty old Glassfish 2.1, can you
> at least update it to the latest 2.1.1?
>
> Thanks.
>
>
> WBR,
> Alexey.
>
> On Jan 24, 2011, at 21:16 , Guillaume wrote:
>
> Hi Alexey,
>
> Unfortunately, we are stuck with glassfish 2.1 for a long time before we
> can change grizzly or glassfish version.
>
> I can't provide a test case, it is the result of a load test. I don't know
> what can produce such "stack traces".
>
> We had a lot of exceptions like these during 10 minutes and it stops.
> Sometimes, some of them appear again but not that much.
>
> I just wanted to know if these errors are known issues or can be the result
> of an overload.
>
> If we can't find out what the problem is, I think our best chance is to
> upgrade glassfish but without certainty. If we can't identify the source of
> the error, then we can't be sure it won't happen again.
>
> If anyone has an idea, don't hesitate to contact me.
>
> Thanks.
>
> William.
>
> On Mon, Jan 24, 2011 at 7:41 PM, Oleksiy Stashok <
> oleksiy.stashok_at_oracle.com> wrote:
>
>> Hi William,
>>
>> did you try this on Glassfish 3.0.1 or 3.1 (latest promoted build)?
>>
>> Can you pls. provide the simple testcase for it?
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>> On Jan 24, 2011, at 15:55 , william.d wrote:
>>
>>
>>> Hello,
>>>
>>> On glassfish 2.1, I have the following errors the server.log files :
>>>
>>> 1./ java.lang.StackOverflowError
>>>
>>> Very strange and scary error but it is printed as a warning in log files.
>>> Does anyone have already seen this error ?
>>>
>>>
>>> WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=172;_ThreadName=httpWorkerThread-8280-466;_Reque
>>> stID=355fd080-200e-4c78-b7c2-9186137e84fb;|
>>> java.lang.StackOverflowError
>>> at
>>> org.apache.tomcat.util.http.MimeHeaders.setValue(MimeHeaders.java:304)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1394)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.flush
>>> (AsynchronousOutputBuffer.java:103)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes
>>> (SocketChannelOutputBuffer.java:145)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.AsynchronousOutputBuffer.realWriteBytes
>>> (AsynchronousOutputBuffer.java:70)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:378)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:815)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:502)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.DefaultProcessorTask.prepareResponse
>>> (DefaultProcessorTask.java:1457)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.action
>>> (DefaultProcessorTask.java:863)
>>> at org.apache.coyote.Response.action(Response.java:235)
>>> at
>>>
>>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:360)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush
>>> (SocketChannelOutputBuffer.java:180)
>>>
>>> 2./ NPE on DefaultAsyncExecutor.postExecute :
>>>
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=171;_ThreadName=httpWorkerThread-8280-86;_Requ
>>> estID=73f0a3ee-5cfa-4bca-a70e-c3938863bc03;|The log message is null.
>>> java.lang.NullPointerException
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.DefaultAsyncExecutor.postExecute
>>> (DefaultAsyncExecutor.java:189)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>>> (AsyncProcessorTask.java:99)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run
>>> (WorkerThreadImpl.java:116)
>>>
>>>
>>> 3./ NPE in CometEngine.handle
>>>
>>> For this one, it seems that the selection key is null (by looking at the
>>> code). Can it be reproduced by a use case ?
>>>
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=170;_ThreadName=httpWorkerThread-8280-43;_Requ
>>> estID=1d902f71-86f0-46f7-9d66-7e9cb47f729f;|The log message is null.
>>> java.lang.NullPointerException
>>> at
>>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.comet.CometEngine.handle
>>> (CometEngine.java:346)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.comet.CometAsyncFilter.doFilter
>>> (CometAsyncFilter.java:84)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.DefaultAsyncExecutor.invokeFilters
>>> (DefaultAsyncExecutor.java:175)
>>> at
>>> com.sun.enterprise.
>>> web.connector.grizzly.async.DefaultAsyncExecutor.interrupt
>>> (DefaultAsyncExecutor.java:153)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask
>>> (AsyncProcessorTask.java:92)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run
>>> (WorkerThreadImpl.java:116)
>>> |#]
>>>
>>>
>>> Errors 2. and 3. seems to be related to async processing.... Do you have
>>> any
>>> idea why a selection key or a processor task could be null ?
>>>
>>> 4. NPE in a ressource bundle.
>>>
>>> For the last one it seems that a bundle is missing but I don't know how
>>> to
>>> finf it in order to fix it.
>>>
>>>
>>> SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=120;_ThreadName=httpWorkerThread-8280-234;_Req
>>>
>>> uestID=84dd1cd9-036c-43fc-930a-af79dc664a6a;|StandardWrapperValve[update]:
>>> Cannot find message associated with key
>>> standardWrapper.acknowledgeExceptio
>>> n
>>> java.lang.NullPointerException
>>> |#]
>>>
>>>
>>> Thanks.
>>>
>>> William.
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Several-weird-errors-under-high-load-with-grizzly-in-glassfish-tp30748059p30748059.html
>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
>