Looks like this is related to
https://glassfish.dev.java.net/issues/show_bug.cgi?id=3960
This will be fixed in 9.1.1.
glassfish_at_javadesktop.org wrote:
> We are facing problems with [b]deadlocks[/b] in glassfish server 9.1 (build b58g-fcs).
>
> Our application leaks connections, so we enabled connection leak detection. But a timer thread of the leak detection collides with a work thread. Clean shutdown was impossible.
>
> Our parameters:
>
> pool size 64 or 80
> Leak Timeout = 300s
> Leak Reclaim enabled
> Lazy Association enabled
> Lazy Connection Enlistment enabled
> Associate with Thread enabled
> Match Connections enabled
>
> <quotation from log>
>
> Found one Java-level deadlock:
> =============================
> "RMI TCP Connection(37)-127.0.0.1":
> waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool),
> which is held by "httpSSLWorkerThread-39080-1"
> "httpSSLWorkerThread-39080-1":
> waiting to lock monitor 0x08ec27f0 (object 0x6289e308, a java.lang.Object),
> which is held by "Timer-6"
> "Timer-6":
> waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool),
> which is held by "httpSSLWorkerThread-39080-1"
>
> Java stack information for the threads listed above:
> ===================================================
> "RMI TCP Connection(37)-127.0.0.1":
> at com.sun.enterprise.resource.AbstractResourcePool.cancelResizerTask(AbstractResourcePool.java:1782)
> - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
> at com.sun.enterprise.resource.PoolManagerImpl.killPool(PoolManagerImpl.java:573)
> at com.sun.enterprise.resource.PoolManagerImpl.killAllPools(PoolManagerImpl.java:867)
> at com.sun.enterprise.server.ApplicationServer.onTermination(ApplicationServer.java:574)
> at com.sun.enterprise.server.Shutdown.startShutdown(Shutdown.java:76)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.invokeShutdownEventListener(AdminEventMulticaster.java:1501)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.handleShutdownEvent(AdminEventMulticaster.java:1490)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:480)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176)
> at com.sun.enterprise.admin.server.core.channel.AdminChannelServer.sendNotification(AdminChannelServer.java:121)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> "httpSSLWorkerThread-39080-1":
> at com.sun.enterprise.resource.AbstractResourcePool.startConnectionLeakTracing(AbstractResourcePool.java:309)
> - waiting to lock <0x6289e308> (a java.lang.Object)
> at com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301)
> at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778)
> - locked <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
> at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652)
> at com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136)
> at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594)
> at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443)
> at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327)
> at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
> at de.denic.guiservices.ejb.AuthServiceManagerBean.getRoles(AuthServiceManagerBean.java:60)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.enterprise.security.SecurityUtil$2.run(SecurityUtil.java:182)
> at java.security.AccessController.doPrivileged(Native Method)
> at com.sun.enterprise.security.application.EJBSecurityManager.doAsPrivileged(EJBSecurityManager.java:1041)
> at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:187)
> at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895)
> at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986)
> at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
> at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:77)
> at $Proxy45.getRoles(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:233)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
> at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
> at de.denic.guiservices.__AuthServiceManager_Remote_DynamicStub.getRoles(de/denic/guiservices/__AuthServiceManager_Remote_DynamicStub.java)
> at de.denic.guiservices._AuthServiceManager_Wrapper.getRoles(de/denic/guiservices/_AuthServiceManager_Wrapper.java)
> at de.denic.mir.controller.Authorisation.getRegAccIdsWhereMemberIsDenied(Authorisation.java:165)
> - locked <0x63f4eb38> (a de.denic.mir.controller.Authorisation)
> at de.denic.mir.controller.MirRegAccController.loadRegAccListTest(MirRegAccController.java:484)
> at de.denic.mir.controller.MirRegAccController.loadRegAccList(MirRegAccController.java:417)
> at de.denic.mir.controller.MirRegAccController.init(MirRegAccController.java:153)
> at org.apache.jsp.de.jsf.mirRegAcc.Init_jsp._jspService(Init_jsp.java from :86)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
> at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
> at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
> at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
> at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192)
> at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404)
> at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855)
> at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703)
> at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542)
> at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474)
> at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:107)
> at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:122)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:352)
> at com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:408)
> at com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:442)
> at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:115)
> at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
> at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
> at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
> at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
> at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
> at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192)
> at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
> at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
> at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361)
> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
> at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
> "Timer-6":
> at com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:1060)
> - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
> at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344)
> - locked <0x6289e308> (a java.lang.Object)
> at com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62)
> at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1543)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Found 1 deadlock.
> [Message sent by forum member 'wjp' (wjp)]
>
> http://forums.java.net/jive/thread.jspa?messageID=267464
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>