Hello Sridatta Viswanath, GlassFish guys
i encountered a deadlock during the the server instance's startup. The
GF version is V2 final release. From the stack trace(attached at
the end of the mail), it seems to be a bug which caused by ConfigBean API,
acting as bellow:
* A sub thread named "RMI TCP Connection(14)-137.172.75.72"£¬ calls the
synchronized API com.sun.enterprise.config.ConfigBean#clone(), and locks
configbeans of config tree in fllowing order:
Domain¡úResources
* The main thread, calls the synchronized API
com.sun.enterprise.config.ConfigBean#getInterceptor(), and lock configbeans
in reverse order:
Resources¡úDomain
Is it a bug? With a greate interest, i looked into the implemention of the
synchronized API, and found the reason. The getInterceptor API performs as a
recursive call in special condition, and locks all configbean object during
its invocation period. The API does not release any lock until it returned.
////////////////////////////////////////////////////////////////////////////
public synchronized ConfigBeanInterceptor getInterceptor() { <------locks this
if (null != _interceptor) {
return _interceptor;
}
ConfigBeanInterceptor cbi = null;
//get interceptor of ctx.
if (null != ctx) {
/**
* Should have used the ConfigContext interface. Too late and
* risky to change the interface.
*/
cbi = ((ConfigContextImpl)ctx).getConfigBeanInterceptor();
} else {
//get interceptor of parent.
ConfigBean parent = (ConfigBean)parent();
/*
Hack :- BaseBean.parent() returns 'this' if this is root.
Added the check (this != parent) to avoid recursion.
*/
if ((null != parent) && (this != parent)) {
cbi = parent.getInterceptor(); <------lock parent
}
}
/*
if (null == cbi) {
cbi = EnvironmentFactory.getEnvironmentFactory().
getConfigEnvironment().getConfigBeanInterceptor();
}*/
//_interceptor = cbi;
return cbi;
}
////////////////////////////////////////////////////////////////////////////
Is the getInterceptor API thread safe? If not, how to fix?
When i tried to fix, i found it is really hard to do. How can i change the
method to be thread safe and do not do any bad influence? Could you give me
some suggestion?
i also tried to sychronize the execution between
com.sun.enterprise.config.ConfigBean#clone() and
com.sun.enterprise.config.ConfigBean#getInterceptor() by creating a new
mutex, but this may cause a new deadlock.Because i do not know who would
call the API.
Besides, i have another question where the sub thread "RMI TCP
Connection(14)-137.172.75.72" is started. Could u do me a favor?
////////////////////////deadlock stack trace showing bellow//////////////////////////////////
Found one Java-level deadlock:
=============================
"RMI TCP Connection(14)-137.172.75.72":
waiting to lock monitor 0x684286c4 (object 0x285ef8a8, a
com.sun.enterprise.config.serverbeans.Resources),
which is held by "main"
"main":
waiting to lock monitor 0x055e2464 (object 0x285d8638, a
com.sun.enterprise.config.serverbeans.Domain),
which is held by "RMI TCP Connection(14)-137.172.75.72"
Java stack information for the threads listed above:
===================================================
"RMI TCP Connection(14)-137.172.75.72":
at com.sun.enterprise.config.ConfigBean.clone(ConfigBean.java:553)
- waiting to lock <0x285ef8a8> (a com.sun.enterprise.config.serverbeans.Resources)
at org.netbeans.modules.schema2beans.BaseBean.clone(BaseBean.java:1294)
at com.sun.enterprise.config.ConfigBean.clone(ConfigBean.java:554)
- locked <0x285d8638> (a com.sun.enterprise.config.serverbeans.Domain)
at com.sun.enterprise.config.impl.ConfigContextImpl.clone(ConfigContextImpl.java:509)
at com.sun.enterprise.admin.event.AdminEventMulticaster.initEventHandler(AdminEventMulticaster.java:523)
at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:453)
at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176)
at com.sun.enterprise.ee.admin.mbeans.ServerRuntimeMBean.forwardEvent(ServerRuntimeMBean.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:375)
at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:358)
at com.sun.enterprise.admin.runtime.BaseRuntimeMBean.invoke(BaseRuntimeMBean.java:462)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:90)
at $Proxy1.invoke(Unknown Source)
at com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:304)
at com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
"main":
at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:779)
- waiting to lock <0x285d8638> (acom.sun.enterprise.config.serverbeans.Domain)
at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:798)
- locked <0x285ef8a8> (a com.sun.enterprise.config.serverbeans.Resources)
at com.sun.enterprise.config.ConfigBean.getInterceptor(ConfigBean.java:798)
- locked <0x286654a0> (a com.sun.enterprise.config.serverbeans.JdbcConnectionPool)
at com.sun.enterprise.config.ConfigBean.postGetAttributeValue(ConfigBean.java:828)
at com.sun.enterprise.config.ConfigBean.getAttributeValue(ConfigBean.java:539)
- locked <0x286654a0> (a com.sun.enterprise.config.serverbeans.JdbcConnectionPool)
at com.sun.enterprise.config.serverbeans.JdbcConnectionPool.getDatasourceClassname(JdbcConnectionPool.java:189)
at com.sun.enterprise.resource.ResourceInstaller.recoverJdbcXAResources(ResourceInstaller.java:512)
at com.sun.enterprise.resource.ResourceInstaller.recoverXAResources(ResourceInstaller.java:331)
at com.sun.enterprise.server.ApplicationLifecycle.onStartup(ApplicationLifecycle.java:205)
at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:464)
at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:120)
at com.sun.enterprise.server.PEMain.run(PEMain.java:436)
at com.sun.enterprise.server.PEMain.main(PEMain.java:359)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.enterprise.server.PELaunch.main(PELaunch.java:412)
Found 1 deadlock.
////////////////////////////end////////////////////////////////////////////////
Have a nice weekend!
Thanks,
Zhang Biyun