Hi Jon,
> This may be more of a Glassfish question, but I thought I'd try here
> first. In Open Wonderland, we are having problems with the web
> server freezing for about a minute at a time under load. We were
> able to identify that some of the freezes were due to Grizzly issue
> #547.
Right, looks similar.
> Unfortunately, even with a fix we still see similar freezes with a
> different stack trace.
Are you sure the Grizzly version used with the GF has the mentioned
#547 fix?
> What we see is that all the http-thread-pool threads get stuck at
> the same spot, as shown in the stack trace at the end of the
> message. What seems to be happening is a slow or disconnected client
> leaves the server with data to flush, but there is a select() in the
> flushChannel() method that can last up to 30 seconds. If enough
> clients stop reading data at the same time, it wedges the entire
> server.
Agree.
>
> I am trying to figure out where to look next. My current thoughts are:
> 1. just increase the size of the http thread pool so the threads
> don't all get stuck. We're using embedded GF, so it's not
> immediately obvious how to do this. Currently we use the default
> pool size of 5 threads.
Oh, I don't know. It's probably better to ask on GF mailing list how
to change embedded GF configuration.
> 2. try to figure out why the servlets are leaving this unflushed
> data, for example due to not calling close() on the response or
> something similar. This seems to be related to the jnlp download
> servlet and / or Jersey.
Could be.
> 3. look at ways to avoid the 30-second timeout in the select() call,
> perhaps by using the existing selector thread. Is there any reason
> this select() is done in a way that might block the http thread,
> instead of as a callback on the selector thread?
By default we're using blocking writes. It works good in most cases
and performs better than async write, though issues like that may
cause problems.
You can try to enable async write using system property:
-Dcom.sun.grizzly.http.asyncwrite.enabled=true
Or to decrease the write timeout pls use system property:
-Dcom.sun.grizzly.writeTimeout = <timeout_millis>
WBR,
Alexey.
>
> I'd appreciate any suggestions.
>
> -Jon
>
> "http-thread-pool-8080-(11)" daemon prio=3 tid=0x0a584000 nid=0x83
> runnable [0xb5725000..0xb5725be0]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
> at
> sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
> at
> sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:
> 69)
> - locked <0xf9ec0ff0> (a sun.nio.ch.Util$1)
> - locked <0xf9ec0fe0> (a java.util.Collections
> $UnmodifiableSet)
> - locked <0xf9ec0dd0> (a sun.nio.ch.DevPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:126)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushChannel(SocketChannelOutputBuffer.java:321)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushBuffer(SocketChannelOutputBuffer.java:392)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .realWriteBytes(SocketChannelOutputBuffer.java:279)
> at com.sun.grizzly.tcp.http11.InternalOutputBuffer
> $OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:884)
> at
> com
> .sun
> .grizzly
> .tcp
> .http11
> .filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:155)
> at
> com
> .sun
> .grizzly
> .tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:
> 640)
> at com.sun.grizzly.tcp.Response.doWrite(Response.java:676)
> at
> org
> .apache
> .catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:420)
> at
> com.sun.grizzly.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
> at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:
> 355)
> at
> org
> .apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:
> 450)
> at
> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:
> 437)
> at
> org
> .apache
> .catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:
> 161)
> at org.jdesktop.deployment.jnlp.servlet.DownloadResponse
> $FileDownloadResponse.sendRespond(DownloadResponse.java:158)
> at
> org
> .jdesktop
> .deployment
> .jnlp
> .servlet.JnlpDownloadServlet.handleRequest(JnlpDownloadServlet.java:
> 127)
> at
> org
> .jdesktop
> .deployment
> .jnlp.servlet.JnlpDownloadServlet.doGet(JnlpDownloadServlet.java:93)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
> 734)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
> 847)
> at
> org
> .apache.catalina.core.StandardWrapper.service(StandardWrapper.java:
> 1523)
> at
> org
> .apache
> .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
> 279)
> at
> org
> .apache
> .catalina.core.StandardContextValve.invoke(StandardContextValve.java:
> 188)
> at
> org
> .apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
> 641)
> at
> com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
> at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke
> (PESessionLockingStandardPipeline.java:85)
> at
> org
> .apache
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
> at
> org
> .apache
> .catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:332)
> at
> org
> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
> 233)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
>