Zhang Biyun,
Shridatta is probably not seeing this e-mail. It's quite possible that
this code isn't thread-safe and like all threading bugs, this bug is
uncovered only in some situations.
The RMI TCP Connection thread is created when you send some
request on the RMI/JMX Channel. It seems to come in the way of
the main thread which isn't done yet (initializing the server).
Can you attach the domain.xml and the entire stack trace in an e-mail?
Thanks,
Kedar Mhaswade
Bring Zhang wrote:
> Hello Sridatta Viswanath, GlassFish guys
>
> i encountered a deadlock during the the server instance's startup. The
> GF version is V2 final release. From the stack trace(attached at
> the end of the mail), it seems to be a bug which caused by ConfigBean API,
> acting as bellow:
>
> * A sub thread named "RMI TCP Connection(14)-137.172.75.72"£¬ calls the
> synchronized API com.sun.enterprise.config.ConfigBean#clone(), and locks
> configbeans of config tree in fllowing order:
>
> Domain¡úResources
>
> * The main thread, calls the synchronized API
> com.sun.enterprise.config.ConfigBean#getInterceptor(), and lock configbeans
> in reverse order:
>
> Resources¡úDomain
>
> Is it a bug? With a greate interest, i looked into the implemention of the
> synchronized API, and found the reason. The getInterceptor API performs as a
> recursive call in special condition, and locks all configbean object during
> its invocation period. The API does not release any lock until it returned.
>
> ////////////////////////////////////////////////////////////////////////////
> public synchronized ConfigBeanInterceptor getInterceptor() { <------locks this
> if (null != _interceptor) {
> return _interceptor;
> }
> ConfigBeanInterceptor cbi = null;
> //get interceptor of ctx.
> if (null != ctx) {
> /**
> * Should have used the ConfigContext interface. Too late and
> * risky to change the interface.
> */
> cbi = ((ConfigContextImpl)ctx).getConfigBeanInterceptor();
> } else {
> //get interceptor of parent.
> ConfigBean parent = (ConfigBean)parent();
> /*
> Hack :- BaseBean.parent() returns 'this' if this is root.
> Added the check (this != parent) to avoid recursion.
> */
> if ((null != parent) && (this != parent)) {
> cbi = parent.getInterceptor(); <------lock parent
> }
> }
> /*
> if (null == cbi) {
> cbi = EnvironmentFactory.getEnvironmentFactory().
> getConfigEnvironment().getConfigBeanInterceptor();
> }*/
> //_interceptor = cbi;
> return cbi;
> }
> ////////////////////////////////////////////////////////////////////////////
>
> Is the getInterceptor API thread safe? If not, how to fix?
> When i tried to fix, i found it is really hard to do. How can i change the
> method to be thread safe and do not do any bad influence? Could you give me
> some suggestion?
>
> i also tried to sychronize the execution between
> com.sun.enterprise.config.ConfigBean#clone() and
> com.sun.enterprise.config.ConfigBean#getInterceptor() by creating a new
> mutex, but this may cause a new deadlock.Because i do not know who would
> call the API.
>
> Besides, i have another question where the sub thread "RMI TCP
> Connection(14)-137.172.75.72" is started. Could u do me a favor?
>
> ////////////////////////deadlock stack trace showing bellow//////////////////////////////////
> Found one Java-level deadlock:
> =============================
> "RMI TCP Connection(14)-137.172.75.72":
> waiting to lock monitor 0x684286c4 (object 0x285ef8a8, a
> com.sun.enterprise.config.serverbeans.Resources),
> which is held by "main"
> "main":
> waiting to lock monitor 0x055e2464 (object 0x285d8638, a
> com.sun.enterprise.config.serverbeans.Domain),
> which is held by "RMI TCP Connection(14)-137.172.75.72"
>
> Java stack information for the threads listed above:
> ===================================================
> "RMI TCP Connection(14)-137.172.75.72":
> at com.sun.enterprise.config.ConfigBean.clone(ConfigBean.java:553)
> - waiting to lock <0x285ef8a8> (a com.sun.enterprise.config.serverbeans.Resources)
> at org.netbeans.modules.schema2beans.BaseBean.clone(BaseBean.java:1294)
> at com.sun.enterprise.config.ConfigBean.clone(ConfigBean.java:554)
> - locked <0x285d8638> (a com.sun.enterprise.config.serverbeans.Domain)
> at com.sun.enterprise.config.impl.ConfigContextImpl.clone(ConfigContextImpl.java:509)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.initEventHandler(AdminEventMulticaster.java:523)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:453)
> at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176)
> at com.sun.enterprise.ee.admin.mbeans.ServerRuntimeMBean.forwardEvent(ServerRuntimeMBean.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:375)
> at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:358)
> at com.sun.enterprise.admin.runtime.BaseRuntimeMBean.invoke(BaseRuntimeMBean.java:462)
> at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
> at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:90)
> at $Proxy1.invoke(Unknown Source)
> at com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:304)
> at com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
> at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
> at sun.rmi.transport.Transport$1.run(Transport.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
> at java.lang.Thread.run(Thread.java:595)
>
> "main":
> at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:779)
> - waiting to lock <0x285d8638> (acom.sun.enterprise.config.serverbeans.Domain)
> at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:798)
> - locked <0x285ef8a8> (a com.sun.enterprise.config.serverbeans.Resources)
> at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:798)
> - locked <0x286654a0> (a com.sun.enterprise.config.serverbeans.JdbcConnectionPool)
> at com.sun.enterprise.config.ConfigBean.postGetAttributeValue(ConfigBean.java:828)
> at com.sun.enterprise.config.ConfigBean.getAttributeValue(ConfigBean.java:539)
> - locked <0x286654a0> (a com.sun.enterprise.config.serverbeans.JdbcConnectionPool)
> at com.sun.enterprise.config.serverbeans.JdbcConnectionPool.getDatasourceClassname(JdbcConnectionPool.java:189)
> at com.sun.enterprise.resource.ResourceInstaller.recoverJdbcXAResources(ResourceInstaller.java:512)
> at com.sun.enterprise.resource.ResourceInstaller.recoverXAResources(ResourceInstaller.java:331)
> at com.sun.enterprise.server.ApplicationLifecycle.onStartup(ApplicationLifecycle.java:205)
> at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:464)
> at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:120)
> at com.sun.enterprise.server.PEMain.run(PEMain.java:436)
> at com.sun.enterprise.server.PEMain.main(PEMain.java:359)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at com.sun.enterprise.server.PELaunch.main(PELaunch.java:412)
>
> Found 1 deadlock.
> ////////////////////////////end////////////////////////////////////////////////
>
> Have a nice weekend!
>
> Thanks,
> Zhang Biyun