I upgraded to "grizzly-http-webserver-1.9.15.jar" yesterday on a site
that is heavily utilized.
Last night it crashed in spectacular fashion, producing a 400MB log file
as it dumped as much as it could to the System.err stream. Fortunately
I have lots of logging setup so you can see what went pear shaped when
it does go down.
Here is a snippet, as this repeats continually for the majority of the
400MB log file.
-- SystemErr --------------------------------
SEVERE: doSelect exception
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at com.sun.grizzly.NIOContext.execute(NIOContext.java:507)
at com.sun.grizzly.NIOContext.execute(NIOContext.java:485)
at
com.sun.grizzly.Controller.handleSelectedKeys(Controller.java:469)
at com.sun.grizzly.Controller.doSelect(Controller.java:337)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:88)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
----------------------------------
this is the extract that repeats, which clearly illustrates it being
stuck in a loop:
-- SystemErr--------------------------------
SEVERE: HTTP Processing error
java.lang.StackOverflowError
at
com.sun.grizzly.util.buf.MessageBytes.equals(MessageBytes.java:351)
at
com.sun.grizzly.http.ProcessorTask.prepareResponse(ProcessorTask.java:1458)
at
com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:938)
at com.sun.grizzly.tcp.Response.action(Response.java:255)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:348)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:367)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:283)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:366)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:831)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:783)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:491)
at
com.sun.grizzly.http.ProcessorTask.prepareResponse(ProcessorTask.java:1536)
at
com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:938)
at com.sun.grizzly.tcp.Response.action(Response.java:255)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:348)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:367)
at
com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:283)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:366)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:831)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:783)
at
com.sun.grizzly.tcp.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:491)
at
com.sun.grizzly.http.ProcessorTask.prepareResponse(ProcessorTask.java:1536)
at
com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:938)
at com.sun.grizzly.tcp.Response.action(Response.java:255)
----------------------------------
Here is a snippet from the Thread Dump at the time of restart
---ThreadDump-------------------------------
All these chaps waiting:
"http8080-WorkerThread(31)" daemon prio=10 tid=0x490a6800 nid=0x11ca
waiting on condition [0x525b1000..0x525b2030]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x632cb430> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)
---ThreadDump-------------------------------
"Thread-8" daemon prio=10 tid=0x0892a800 nid=0x55c5 runnable
[0x52ede000..0x52edf030]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x62ebb530> (a sun.nio.ch.Util$1)
- locked <0x62ebb540> (a java.util.Collections$UnmodifiableSet)
- locked <0x62ebb4f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:492)
at com.sun.grizzly.Controller.doSelect(Controller.java:328)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:88)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
----------------------------------
H E L P! :)