Bhakti Mehta wrote On 02/27/07 02:55 PM,:
> Thanks Mark this was helpful.Also Kedar suggested trying asadmin
> start-domain --verbose domain1 and doing control break to get the
> thread dump.
> Here is the thread dump. Jan is looking into it
Thanks, Bhakti.
From looking at the thread dumps, there seems to be a problem with
JDK's java.util.concurrent.locks.ReentrantReadWriteLock(). I had run
into a similar issue with this kind of lock (in a different code path)
earlier, but was able to work around it.
The root cause that's triggering the issue is a problem with the
admingui:
Exception starting filter HelpWindowFilter
java.lang.ClassNotFoundException:
com.sun.enterprise.tools.admingui.servlet.HelpWindowFilter
When the admingui attempts to log this exception, it tries to acquire a
logger, as follows
(see org.apache.catalina.core.ContainerBase.getLogger()):
public Logger getLogger() {
try {
readLock.lock();
if (logger != null)
return (logger);
} finally {
readLock.unlock();
}
if (parent != null)
return (parent.getLogger());
return (null);
}
Since the admingui's "logger" instance variable is null,
parent.getLogger() is invoked, which uses the same getLogger() impl as
above (in this case, "parent" is the virtual server on which the
admingui has been deployed). After calling the
readLock.lock();
of its parent virtual server, the admingui starts hanging:
"pool-1-thread-7" prio=6 tid=0x270fc368 nid=0x17d4 waiting on condition
[0x2857f000..0x2857fb68]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:681)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:809)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1124)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:423)
at
org.apache.catalina.core.ContainerBase.getLogger(ContainerBase.java:455)
at
org.apache.catalina.core.ContainerBase.getLogger(ContainerBase.java:463)
at
org.apache.catalina.core.ApplicationContext.log(ApplicationContext.java:805)
at
org.apache.catalina.core.ApplicationContextFacade.log(ApplicationContextFacade.java:294)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4340)
After removing the lock (which, of course, is not an option), Bhakti
was able to start her domain, and was able to see the
java.lang.ClassNotFoundException:
com.sun.enterprise.tools.admingui.servlet.HelpWindowFilter
which has triggered the locking problem in the first place (if there had
not been
any exception, there would not have been any need to acquire any logger).
In summary, there are 2 issues:
- Admin team needs to fix the java.lang.ClassNotFoundException
- We need to further investigate why JDK's ReentrantReadWriteLock may
cause hanging.
Jan