I am running a number of Glassfish V3 servers on both Windows and Linux boxes. All servers run a single application, consisting of a JaxXWS based web service and some servlets, both of which utilize EJB3 for persisting and accessing a mysql database configured as a JNDI datasource.
On all server instances I observe the following issue:
Approx 2 - 3 times a day, we will observe a "broken pipe" exception (see below), which results in the whole application becoming unresponsive for up to 2 minutes: Calls to both servlets and the web service will simply time out. The logs show no incoming requests, packet sniffing on the web port shows no traffic.
All the while, ssh or even access to the admin console or other services on the box are not effected.
The issue seems independent of the sever load (average load is one ws call every 7 seconds, one servlet request every 5 sec), calls contain very little data.
After a while (12 sec to several minutes) the app will resume normal operation. There is no observable change in application behavior (slowdown, etc) before or after the "outage", and no other error messages can be observe.
In short, it looks like the web container simply "pauses"....
Memory settings as well as different pool sizes for DB and number of accepting threads does not show any change in behavior.
The same issue could not be observed in older glassfish versions.
[#|2010-03-18T13:05:50.621-0700|WARNING|glassfishv3.0|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=30;_ThreadName=Thread-1;|StandardWrapperValve[PBDEnergyConsumptionCollectorService]:
PWC1406: Servlet.service() for servlet
PBDEnergyConsumptionCollectorService threw exception
javax.servlet.ServletException
at org.glassfish.webservices.JAXWSServlet.doGet(JAXWSServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:332)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:233)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: java.io.IOException: Broken pipe
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:280)
at com.sun.xml.ws.transport.http.HttpAdapter.publishWSDL(HttpAdapter.java:583)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:252)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.publishWSDL(ServletAdapter.java:157)
at org.glassfish.webservices.JAXWSServlet.doGet(JAXWSServlet.java:213)
... 26 more
Caused by: com.ctc.wstx.exc.WstxIOException: java.io.IOException: Broken pipe
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1687)
at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:278)
... 30 more
Caused by: org.apache.catalina.connector.ClientAbortException:
java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:382)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:351)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1685)
... 32 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:106)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:321)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:392)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:370)
at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1174)
at com.sun.grizzly.tcp.Response.action(Response.java:266)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:378)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:351)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1685)
at com.ctc.wstx.sw.BaseStreamWriter.writeEndDocument(BaseStreamWriter.java:585)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:270)
... 30 more
[Message sent by forum member 'maikhassel']
http://forums.java.net/jive/thread.jspa?messageID=392790