This may be more of a Glassfish question, but I thought I'd try here first.
In Open Wonderland, we are having problems with the web server freezing for
about a minute at a time under load. We were able to identify that some of
the freezes were due to Grizzly issue #547. Unfortunately, even with a fix
we still see similar freezes with a different stack trace.
What we see is that all the http-thread-pool threads get stuck at the same
spot, as shown in the stack trace at the end of the message. What seems to
be happening is a slow or disconnected client leaves the server with data to
flush, but there is a select() in the flushChannel() method that can last up
to 30 seconds. If enough clients stop reading data at the same time, it
wedges the entire server.
I am trying to figure out where to look next. My current thoughts are:
1. just increase the size of the http thread pool so the threads don't all
get stuck. We're using embedded GF, so it's not immediately obvious how to
do this. Currently we use the default pool size of 5 threads.
2. try to figure out why the servlets are leaving this unflushed data, for
example due to not calling close() on the response or something similar.
This seems to be related to the jnlp download servlet and / or Jersey.
3. look at ways to avoid the 30-second timeout in the select() call, perhaps
by using the existing selector thread. Is there any reason this select() is
done in a way that might block the http thread, instead of as a callback on
the selector thread?
I'd appreciate any suggestions.
-Jon
"http-thread-pool-8080-(11)" daemon prio=3 tid=0x0a584000 nid=0x83 runnable
[0xb5725000..0xb5725be0]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf9ec0ff0> (a sun.nio.ch.Util$1)
- locked <0xf9ec0fe0> (a java.util.Collections$UnmodifiableSet)
- locked <0xf9ec0dd0> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:126)
at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:321)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:392)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:279)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:884)
at
com.sun.grizzly.tcp.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:155)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:640)
at com.sun.grizzly.tcp.Response.doWrite(Response.java:676)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:420)
at
com.sun.grizzly.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:355)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:450)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:437)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:161)
at
org.jdesktop.deployment.jnlp.servlet.DownloadResponse$FileDownloadResponse.sendRespond(DownloadResponse.java:158)
at
org.jdesktop.deployment.jnlp.servlet.JnlpDownloadServlet.handleRequest(JnlpDownloadServlet.java:127)
at
org.jdesktop.deployment.jnlp.servlet.JnlpDownloadServlet.doGet(JnlpDownloadServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
at
org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:332)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:233)
at
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
at
com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
at
com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
at
com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at
com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:619)