We are facing problems with [b]deadlocks[/b] in glassfish server 9.1 (build b58g-fcs).
Our application leaks connections, so we enabled connection leak detection. But a timer thread of the leak detection collides with a work thread. Clean shutdown was impossible.
Our parameters:
pool size 64 or 80
Leak Timeout = 300s
Leak Reclaim enabled
Lazy Association enabled
Lazy Connection Enlistment enabled
Associate with Thread enabled
Match Connections enabled
<quotation from log>
Found one Java-level deadlock:
=============================
"RMI TCP Connection(37)-127.0.0.1":
waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool),
which is held by "httpSSLWorkerThread-39080-1"
"httpSSLWorkerThread-39080-1":
waiting to lock monitor 0x08ec27f0 (object 0x6289e308, a java.lang.Object),
which is held by "Timer-6"
"Timer-6":
waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool),
which is held by "httpSSLWorkerThread-39080-1"
Java stack information for the threads listed above:
===================================================
"RMI TCP Connection(37)-127.0.0.1":
at com.sun.enterprise.resource.AbstractResourcePool.cancelResizerTask(AbstractResourcePool.java:1782)
- waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
at com.sun.enterprise.resource.PoolManagerImpl.killPool(PoolManagerImpl.java:573)
at com.sun.enterprise.resource.PoolManagerImpl.killAllPools(PoolManagerImpl.java:867)
at com.sun.enterprise.server.ApplicationServer.onTermination(ApplicationServer.java:574)
at com.sun.enterprise.server.Shutdown.startShutdown(Shutdown.java:76)
at com.sun.enterprise.admin.event.AdminEventMulticaster.invokeShutdownEventListener(AdminEventMulticaster.java:1501)
at com.sun.enterprise.admin.event.AdminEventMulticaster.handleShutdownEvent(AdminEventMulticaster.java:1490)
at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:480)
at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176)
at com.sun.enterprise.admin.server.core.channel.AdminChannelServer.sendNotification(AdminChannelServer.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"httpSSLWorkerThread-39080-1":
at com.sun.enterprise.resource.AbstractResourcePool.startConnectionLeakTracing(AbstractResourcePool.java:309)
- waiting to lock <0x6289e308> (a java.lang.Object)
at com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301)
at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778)
- locked <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652)
at com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136)
at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594)
at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443)
at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
at de.denic.guiservices.ejb.AuthServiceManagerBean.getRoles(AuthServiceManagerBean.java:60)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.enterprise.security.SecurityUtil$2.run(SecurityUtil.java:182)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.enterprise.security.application.EJBSecurityManager.doAsPrivileged(EJBSecurityManager.java:1041)
at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:187)
at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:77)
at $Proxy45.getRoles(Unknown Source)
at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:233)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at de.denic.guiservices.__AuthServiceManager_Remote_DynamicStub.getRoles(de/denic/guiservices/__AuthServiceManager_Remote_DynamicStub.java)
at de.denic.guiservices._AuthServiceManager_Wrapper.getRoles(de/denic/guiservices/_AuthServiceManager_Wrapper.java)
at de.denic.mir.controller.Authorisation.getRegAccIdsWhereMemberIsDenied(Authorisation.java:165)
- locked <0x63f4eb38> (a de.denic.mir.controller.Authorisation)
at de.denic.mir.controller.MirRegAccController.loadRegAccListTest(MirRegAccController.java:484)
at de.denic.mir.controller.MirRegAccController.loadRegAccList(MirRegAccController.java:417)
at de.denic.mir.controller.MirRegAccController.init(MirRegAccController.java:153)
at org.apache.jsp.de.jsf.mirRegAcc.Init_jsp._jspService(Init_jsp.java from :86)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404)
at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474)
at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:107)
at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:122)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:352)
at com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:408)
at com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:442)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:115)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
"Timer-6":
at com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:1060)
- waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344)
- locked <0x6289e308> (a java.lang.Object)
at com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62)
at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1543)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Found 1 deadlock.
[Message sent by forum member 'wjp' (wjp)]
http://forums.java.net/jive/thread.jspa?messageID=267464