We are in a situation where we find our Glassfish 3.0.1 instance permanently
running at 100% CPU, with heap memory use normal. A thread dump (successive
times) shows that two of our Glassfish threads are spinning indefinitely in
the GZIP compression phase. There appear to be others who are seeing
similar behaviors (but with no answers) and it is starting to look like a
Glassfish GZIP bug. We are simply using the @GZIP annotation on our web
service, and have enabled gzip,deflate in the network/http listener in the
glassfish admin console.
Normally all is well (we see gzipped payloads), so this is NOT the issue
mentioned here:
http://www.java.net/forum/topic/glassfish/glassfish/gzip-compression-glassfish-3-doesnt-work?force=899
[1] Others are seeing similar (maybe exactly the same) problems:
https://forums.oracle.com/forums/thread.jspa?messageID=4627097 [2] and there
are several "clues" in various other stack dumps in a google search on (for
example "Deflater.java:306") that suggest that either this is a
"normal" situation (like socket reads being marked as runnable - I think
not in this case), or that there is some systematic problem with GZIP support
in the JVM. I have found no answers that seem to address this specific
problem.
We have two threads that are in basically the same situation, with different
locks but otherwise stuck at the same place:
"http-thread-pool-8080-(18)" - Thread t_at_950
java.lang.Thread.State: RUNNABLE at
java.util.zip.Deflater.deflateBytes(Native Method) at
java.util.zip.Deflater.deflate(Deflater.java:306) - locked <21b0c5> (a
java.util.zip.ZStreamRef) at
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:159) at
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118) at
java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72) - locked
<132ba84> (a java.util.zip.GZIPOutputStream) at
org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:46)
at
org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
at
org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:46)
at
org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89)
at
org.apache.cxf.io.AbstractThresholdOutputStream.close(AbstractThresholdOutputStream.java:100)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at
org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:619)
at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)
- locked <1ab9b7e> (a org.apache.cxf.phase.PhaseInterceptorChain) at
org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:99)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:315)
- locked <16129e9> (a org.apache.cxf.phase.PhaseInterceptorChain) at
org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:77)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)
- locked <1649a52> (a org.apache.cxf.phase.PhaseInterceptorChain) at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:113)
at
org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:97)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:461)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:149)
at
org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at
org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
at
com.cablelabs.ipr.context.filter.AcceptServletFilter.doFilter(AcceptServletFilter.java:40)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97) at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
at
org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:322)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:226)
at
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693) at
com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954) at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69) at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: -
locked <1cfb6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
I would love to have someone smarter and more experienced than me to shed
some light on this.
[1]
http://www.java.net/forum/topic/glassfish/glassfish/gzip-compression-glassfish-3-doesnt-work?force=899
[2]
https://forums.oracle.com/forums/thread.jspa?messageID=4627097
--
[Message sent by forum member 'steveglennon']
View Post: http://forums.java.net/node/853023