Hi, We are running: Sun GlassFish Communications Server 2.0 ((v2.1
Patch18)(9.1_02 Patch24)) (build b02-p12) Java version 1.6.0_22-b04 We have
expreienced system freezing (response timeout) issue in a cluster with 10+
instances, the only way to fix it is to restart the cluster. We are
constantly taking thread dumps. Now when it happened again and according to
the thread dump we see that the Glassfish Mbean-related code is the
bottleneck (locking com.sun.jmx.interceptor.DefaultMBeanServerInterceptor
which will cause many other threads waiting for the same object) but the
thread holding the lock is not blocked and only does some simple operation:
Sample 1: Sample thread dump on instance 1 for the thread responsible for the
bottleneck: "httpSSLWorkerThread-8080-120" daemon prio=10
tid=0x00002aabf999d800 nid=0x41c4 runnable [0x000000007e407000]
java.lang.Thread.State: RUNNABLE at
java.lang.String.substring(String.java:1939) at
javax.management.ObjectName.getDomain(ObjectName.java:1589) at
com.sun.jmx.mbeanserver.Repository.retrieveNamedObject(Repository.java:326)
at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:492) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabbd9d290> (a
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)'
... ... Sample 2: Smaple thread dump on instance 2 for the thread responsible
for the bottleneck (taken at the same time as Sample 1):
"httpSSLWorkerThread-8080-23" daemon prio=10 tid=0x00002aabf8e69000
nid=0x7968 runnable [0x000000006b8dc000] java.lang.Thread.State: RUNNABLE at
java.lang.String.substring(String.java:1939) at
javax.management.ObjectName.getDomain(ObjectName.java:1589) at
com.sun.jmx.mbeanserver.Repository.retrieveNamedObject(Repository.java:326)
at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:492) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabaf83aa0> (a
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
... ... Sample 3: Sample thread dump on instance 1 (taken 1 minute later) for
the thread responsible for the bottleneck (now its another thread):
"httpSSLWorkerThread-8080-98" daemon prio=10 tid=0x00002aabf3d0b800
nid=0x41ae waiting for monitor entry [0x000000007c2e6000]
java.lang.Thread.State: BLOCKED (on object monitor) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabbd9d290> (a
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:833)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at
com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
... ... Sample 4: Sample thread dump on instance 2 (taken at the same time as
Sample 3) for the thread responsible for the bottleneck (now its another
thread): "httpSSLWorkerThread-8080-50" daemon prio=10 tid=0x00002aabf9022000
nid=0x7983 runnable [0x000000006d3f8000] java.lang.Thread.State: RUNNABLE at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabaf83aa0> (a
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
... ... This pattern continues until we restart the cluster. I just wonder
why all those instances got thread contention on Mbean at the same time, is
there any network issue towards the DAS when synchronizing with the
MbeanServer? Or could it be caused by some other issues? In the DAS server
log I do see no responding exception when trying to reaching the MbeanServer
on the instances, but this error is printed out 30 minutes after the thread
contention started:
[#|2013-02-07T05:46:23.404-0200|WARNING|sun-glassfish-comms-server2.0|javax.enterprise.system.stream.err|_ThreadID=37;_ThreadName=httpSSLWorkerThread-4848-2;_RequestID=b139b136-ee14-4ff8-94a2-d6dca64a1cec;|java.security.PrivilegedActionException:
java.lang.reflect.InvocationTargetException at
java.security.AccessController.doPrivileged(Native Method) at
com.sun.enterprise.jbi.AppServerContextImpl.getMBeanServerConnection(AppServerContextImpl.java:186)
at
com.sun.jbi.framework.sun.SunASPlatformContext.getMBeanServerConnection(SunASPlatformContext.java:206)
at
com.sun.jbi.management.facade.Facade.getMBeanServerConnection(Facade.java:218)
at
com.sun.jbi.management.facade.Facade.mbeanRegistrationCheck(Facade.java:769)
at
com.sun.jbi.management.facade.Facade.invokeRemoteOperation(Facade.java:277)
at
com.sun.jbi.management.facade.Facade.getComponentLifeCycleMBeanName(Facade.java:1204)
at
com.sun.jbi.management.facade.Facade.getComponentStateOnInstance(Facade.java:576)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnInstance(ComponentLifeCycle.java:324)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnCluster(ComponentLifeCycle.java:342)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnTarget(ComponentLifeCycle.java:308)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentState(ComponentLifeCycle.java:176)
at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) at
javax.management.StandardMBean.getAttribute(StandardMBean.java:358) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnAllTargets(ComponentLifeCycle.java:762)
at
com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentState(ComponentLifeCycle.java:180)
at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) at
javax.management.StandardMBean.getAttribute(StandardMBean.java:358) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
at
com.sun.esb.management.base.services.AbstractServiceMBeansImpl.getMBeanAttribute(AbstractServiceMBeansImpl.java:663)
at
com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.toUiComponentInfo(AbstractListStateServiceMBeansImpl.java:664)
at
com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.toUiComponentInfoList(AbstractListStateServiceMBeansImpl.java:606)
at
com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.listComponents(AbstractListStateServiceMBeansImpl.java:272)
at
com.sun.esb.management.impl.runtime.RuntimeManagementServiceMBeanImpl.listBindingComponents(RuntimeManagementServiceMBeanImpl.java:121)
at sun.reflect.GeneratedMethodAccessor1134.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at
javax.management.StandardMBean.invoke(StandardMBean.java:391) at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at
com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
at
com.sun.esb.management.base.services.AbstractServiceImpl.invokeMBeanOperation(AbstractServiceImpl.java:330)
at
com.sun.esb.management.impl.runtime.RuntimeManagementServiceImpl.listBindingComponents(RuntimeManagementServiceImpl.java:144)
at
com.sun.jbi.ui.client.JBIAdminCommandsClientImpl.listBindingComponents(JBIAdminCommandsClientImpl.java:581)
at
com.sun.jbi.ui.client.JBIAdminCommandsClientImpl.listBindingComponents(JBIAdminCommandsClientImpl.java:549)
at com.sun.jbi.jsf.bean.ListBean.getListBindingComponents(ListBean.java:486)
at
com.sun.jbi.jsf.util.JBITreeAdaptorImpl.getChildTreeNodeObjects(JBITreeAdaptorImpl.java:175)
at
com.sun.jsftemplating.component.factory.tree.DynamicTreeNodeFactory.processNode(DynamicTreeNodeFactory.java:178)
at
com.sun.jsftemplating.component.factory.tree.DynamicTreeNodeFactory.create(DynamicTreeNodeFactory.java:109)
at
com.sun.jsftemplating.component.ComponentUtil.createChildComponent(ComponentUtil.java:397)
at
com.sun.jsftemplating.layout.descriptors.LayoutComponent.getChild(LayoutComponent.java:278)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:538)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at
com.sun.jsftemplating.layout.LayoutViewHandler.createView(LayoutViewHandler.java:237)
at
com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:208)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) at
com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:102)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118) at
com.sun.faces.extensions.avatar.lifecycle.PartialTraversalLifecycle.execute(PartialTraversalLifecycle.java:80)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265) at
com.sun.enterprise.tools.admingui.servlet.DelayedInitFacesServlet.service(DelayedInitFacesServlet.java:89)
at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:333)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291) at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.GeneratedMethodAccessor346.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
com.sun.enterprise.jbi.AppServerContextImpl$2.run(AppServerContextImpl.java:189)
... 112 more Caused by:
com.sun.enterprise.admin.servermgmt.InstanceException: Not responding:
service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at
com.sun.enterprise.ee.admin.clientreg.InstanceRegistry.getInstanceConnection(InstanceRegistry.java:163)
... 116 more Caused by:
com.sun.enterprise.ee.admin.servermgmt.AgentException: Not responding:
service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at
com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.getConnection(JMXConnectorRegistry.java:346)
at
com.sun.enterprise.ee.admin.clientreg.InstanceRegistry.getInstanceConnection(InstanceRegistry.java:161)
... 116 more Caused by: java.io.IOException: Not responding:
service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at
com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connectToServer(JMXConnectorRegistry.java:418)
at
com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connect(JMXConnectorRegistry.java:371)
at
com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connect(JMXConnectorRegistry.java:224)
at
com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.getConnection(JMXConnectorRegistry.java:324)
... 117 more |#]
--
[Message sent by forum member 'yhzs8']
View Post: http://forums.java.net/node/894539