dev@glassfish.java.net

Re: intermittent deadlock in GF due to logging

From: Amy Roh <Amelia.Roh_at_Sun.COM>
Date: Fri, 10 Apr 2009 11:45:10 -0700

Just want to share that I still observe a deadlock using the latest
workspace from this morning on mac (no custom logging handler has been
added)

=========

startDomainUnix:
      [exec]
      [exec] Launching Watchdog in a separate JVM. The Domain is now
remote-restart enabled.
      [exec]
      [exec] Domain (domain1) did not respond in 90 seconds. It means it
is still coming up
      [exec] or it has failed to come up. Check server.log for details.

BUILD FAILED
/Users/Amy/glassfish-v3/v3/appserv-tests/config/run.xml:32: The
following error occurred while executing this line:
/Users/Amy/glassfish-v3/v3/appserv-tests/config/run.xml:38: exec returned: 1

Total time: 1 minute 54 seconds
in flushAll , creating new testSuiteHash
bash-3.2$ jps
20756 Jps
20752 ASWatchdog
20753 ASMain
132
bash-3.2$ jstack 20753
2009-04-10 11:39:29
Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.6.0_07-b06-57
mixed mode):

"Attach Listener" daemon prio=9 tid=0x0000000101a93000 nid=0x1347d9000
waiting on condition [0x0000000000000000..0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"RMI TCP Accept-8686" daemon prio=5 tid=0x0000000101a5b800
nid=0x1346d6000 runnable [0x00000001346d5000..0x00000001346d5ad0]
    java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x0000000110852348> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:637)

"AutoDeployer" daemon prio=5 tid=0x0000000101a52000 nid=0x1345d3000
waiting for monitor entry [0x00000001345d2000..0x00000001345d2ad0]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.logging.LogManager.getLogger(LogManager.java:550)
        - waiting to lock <0x000000010facca78> (a java.util.logging.LogManager)
        at com.sun.logging.LogDomains.getLogger(LogDomains.java:246)
        at
org.glassfish.deployment.autodeploy.AutoDeployDirectoryScanner.<clinit>(AutoDeployDirectoryScanner.java:66)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.deployAll(AutoDeployer.java:385)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:343)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:331)
        at
org.glassfish.deployment.autodeploy.AutoDeployService$1.run(AutoDeployService.java:200)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

"Thread-6" prio=5 tid=0x0000000101a51800 nid=0x1344d0000 in
Object.wait() [0x00000001344cf000..0x00000001344cfad0]
    java.lang.Thread.State: RUNNABLE
        at com.sun.grizzly.tcp.Request.<init>(Request.java:185)
        at com.sun.grizzly.http.ProcessorTask.initialize(ProcessorTask.java:475)
        at
com.sun.grizzly.http.SelectorThread.rampUpProcessorTask(SelectorThread.java:916)
        at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1008)
        at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1088)
        at
com.sun.enterprise.v3.services.impl.GrizzlyServiceListener.start(GrizzlyServiceListener.java:84)
        at
com.sun.enterprise.v3.services.impl.GrizzlyProxy$1.run(GrizzlyProxy.java:209)

"Thread-5" prio=5 tid=0x0000000101a4d800 nid=0x1343cd000 waiting for
monitor entry [0x00000001343cc000..0x00000001343ccad0]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.logging.Logger.getLogger(Logger.java:274)
        - waiting to lock <0x00000001252f1550> (a java.lang.Class for
java.util.logging.Logger)
        at com.sun.grizzly.util.http.Cookies.<clinit>(Cookies.java:76)
        at com.sun.grizzly.tcp.Request.<init>(Request.java:185)
        at com.sun.grizzly.http.ProcessorTask.initialize(ProcessorTask.java:475)
        at
com.sun.grizzly.http.SelectorThread.rampUpProcessorTask(SelectorThread.java:916)
        at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1008)
        at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1088)
        at
com.sun.enterprise.v3.services.impl.GrizzlyServiceListener.start(GrizzlyServiceListener.java:84)
        at
com.sun.enterprise.v3.services.impl.GrizzlyProxy$1.run(GrizzlyProxy.java:209)

"DynamicReloader" daemon prio=5 tid=0x0000000101a09000 nid=0x1342ca000
in Object.wait() [0x00000001342c9000..0x00000001342c9ad0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000001107b9530> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000001107b9530> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"pool-1-thread-4" prio=5 tid=0x00000001019f9800 nid=0x13419e000 waiting
on condition [0x000000013419d000..0x000000013419dad0]
    java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000001107b9640> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)

"Thread-3" daemon prio=5 tid=0x00000001019f2000 nid=0x134001000 waiting
for monitor entry [0x0000000133fff000..0x0000000134000ad0]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.logging.LogManager.getLogger(LogManager.java:550)
        - waiting to lock <0x000000010facca78> (a java.util.logging.LogManager)
        at java.util.logging.Logger.getLogger(Logger.java:275)
        - locked <0x00000001252f1550> (a java.lang.Class for
java.util.logging.Logger)
        at com.sun.jmx.remote.util.ClassLogger.<init>(ClassLogger.java:37)
        at
javax.management.remote.JMXConnectorServerFactory.<clinit>(JMXConnectorServerFactory.java:182)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread$RMIConnectorStarter.startRMIConnector(ConnectorStartupService.java:282)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread.startConnector(ConnectorStartupService.java:309)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread.run(ConnectorStartupService.java:339)

"pool-3-thread-1" daemon prio=5 tid=0x0000000101992000 nid=0x133e99000
waiting for monitor entry [0x0000000133e97000..0x0000000133e98ad0]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.logging.Logger.getLogger(Logger.java:274)
        - waiting to lock <0x00000001252f1550> (a java.lang.Class for
java.util.logging.Logger)
        at java.util.logging.LogManager.addLogger(LogManager.java:399)
        - locked <0x000000010facca78> (a java.util.logging.LogManager)
        at com.sun.logging.LogDomains.getLogger(LogDomains.java:331)
        at
com.sun.enterprise.util.i18n.StringManagerBase.<clinit>(StringManagerBase.java:101)
        at
com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.<clinit>(ResourceRecoveryManagerImpl.java:89)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at com.sun.hk2.component.ConstructorWomb.create(ConstructorWomb.java:72)
        at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:76)
        at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
        - locked <0x0000000110853138> (a com.sun.hk2.component.SingletonInhabitant)
        at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
        at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
        at
com.sun.enterprise.v3.server.AppServerStartup$2.run(AppServerStartup.java:197)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)

"Thread-1" prio=5 tid=0x0000000101991000 nid=0x133c3b000 waiting on
condition [0x0000000133c3a000..0x0000000133c3aad0]
    java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000001106c4240> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at
com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:479)
        at
com.sun.enterprise.server.logging.GFFileHandler$3.run(GFFileHandler.java:238)

"pool-2-thread-1" daemon prio=5 tid=0x000000010198c000 nid=0x133b08000
waiting on condition [0x0000000133b07000..0x0000000133b07ad0]
    java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000001106c4428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
        at java.lang.Thread.run(Thread.java:637)

"GlassFish Kernel Main Thread" prio=5 tid=0x0000000101982800
nid=0x133d96000 waiting for monitor entry
[0x0000000133d93000..0x0000000133d95ad0]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at
java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:871)
        - waiting to lock <0x000000010facca78> (a java.util.logging.LogManager)
        at java.util.logging.LogManager.access$900(LogManager.java:129)
        at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:994)
        at java.util.logging.Logger.log(Logger.java:468)
        at java.util.logging.Logger.doLog(Logger.java:494)
        at java.util.logging.Logger.log(Logger.java:517)
        at
com.sun.enterprise.v3.admin.adapter.AdminConsoleAdapter.setStateMsg(AdminConsoleAdapter.java:368)
        at
com.sun.enterprise.v3.admin.adapter.AdminConsoleAdapter.initState(AdminConsoleAdapter.java:445)
        at
com.sun.enterprise.v3.admin.adapter.AdminConsoleAdapter.init(AdminConsoleAdapter.java:431)
        at
com.sun.enterprise.v3.admin.adapter.AdminConsoleAdapter.postConstruct(AdminConsoleAdapter.java:385)
        at com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:160)
        at com.sun.hk2.component.ConstructorWomb$1.run(ConstructorWomb.java:90)
        at java.security.AccessController.doPrivileged(Native Method)
        at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:87)
        at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:77)
        at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
        - locked <0x00000001056f0918> (a com.sun.hk2.component.SingletonInhabitant)
        at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
        at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
        at org.jvnet.hk2.component.Habitat$1.get(Habitat.java:252)
        at java.util.AbstractList$Itr.next(AbstractList.java:345)
        at
com.sun.enterprise.v3.services.impl.GrizzlyService.registerNetworkProxy(GrizzlyService.java:400)
        at
com.sun.enterprise.v3.services.impl.GrizzlyService.registerNetworkProxy(GrizzlyService.java:392)
        at
com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:293)
        at com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:160)
        at com.sun.hk2.component.ConstructorWomb$1.run(ConstructorWomb.java:90)
        at java.security.AccessController.doPrivileged(Native Method)
        at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:87)
        at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:77)
        at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
        - locked <0x00000001106c4748> (a com.sun.hk2.component.SingletonInhabitant)
        at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
        at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
        at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:212)
        at
com.sun.enterprise.v3.server.AppServerStartup$1.run(AppServerStartup.java:124)

"pool-1-thread-3" prio=5 tid=0x0000000101969000 nid=0x133a05000 waiting
on condition [0x0000000133a04000..0x0000000133a04ad0]
    java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000001106082a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)

"pool-1-thread-2" prio=5 tid=0x0000000101965800 nid=0x1338e3000 waiting
on condition [0x00000001338e2000..0x00000001338e2ad0]
    java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000001106084f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)

"pool-1-thread-1" prio=5 tid=0x0000000101964800 nid=0x1337e0000 waiting
on condition [0x00000001337df000..0x00000001337dfad0]
    java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0000000110608750> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)

"{felix.fileinstall.poll=5000,
felix.fileinstall.bundles.new.start=false,
felix.fileinstall.dir=/Users/Amy/v3-4-10/v3/distributions/glassfish/target/glassfish/modules/,
felix.fileinstall.debug=1}" daemon prio=5 tid=0x00000001018fe800
nid=0x13333f000 waiting on condition
[0x000000013333e000..0x000000013333ead0]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.felix.fileinstall.DirectoryWatcher.run(DirectoryWatcher.java:122)

"Configuration Updater" daemon prio=5 tid=0x00000001018fe000
nid=0x13323c000 in Object.wait() [0x000000013323b000..0x000000013323bad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000001101b59e8> (a java.util.LinkedList)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:67)
        - locked <0x00000001101b59e8> (a java.util.LinkedList)

"telnetconsole.Listener" daemon prio=5 tid=0x00000001018f3000
nid=0x133139000 runnable [0x0000000133138000..0x0000000133138ad0]
    java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x00000001101b5ad0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.felix.shell.remote.Listener$Acceptor.run(Listener.java:137)
        at java.lang.Thread.run(Thread.java:637)

"FelixPackageAdmin" daemon prio=5 tid=0x00000001018e8000 nid=0x133036000
in Object.wait() [0x0000000133035000..0x0000000133035ad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000001101b5d50> (a
org.apache.felix.framework.PackageAdminImpl)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.PackageAdminImpl.run(PackageAdminImpl.java:314)
        - locked <0x00000001101b5d50> (a
org.apache.felix.framework.PackageAdminImpl)
        at java.lang.Thread.run(Thread.java:637)

"FelixStartLevel" daemon prio=5 tid=0x00000001018e7000 nid=0x132f33000
in Object.wait() [0x0000000132f32000..0x0000000132f32ad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000001101b6538> (a java.util.ArrayList)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:241)
        - locked <0x00000001101b6538> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:637)

"FelixDispatchQueue" daemon prio=5 tid=0x000000010186b800
nid=0x132756000 in Object.wait() [0x0000000132755000..0x0000000132755ad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000010fb3f7c8> (a java.util.ArrayList)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:875)
        - locked <0x000000010fb3f7c8> (a java.util.ArrayList)
        at
org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:50)
        at
org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:102)
        at java.lang.Thread.run(Thread.java:637)

"OSGi Framework Launcher" daemon prio=5 tid=0x0000000101887800
nid=0x131fe6000 in Object.wait() [0x0000000131fe4000..0x0000000131fe5ad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000001101b9e40> (a
org.apache.felix.framework.util.ThreadGate)
        at org.apache.felix.framework.util.ThreadGate.await(ThreadGate.java:48)
        - locked <0x00000001101b9e40> (a
org.apache.felix.framework.util.ThreadGate)
        at org.apache.felix.framework.Felix.waitForStop(Felix.java:785)
        at org.apache.felix.main.Main.main(Main.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
com.sun.enterprise.glassfish.bootstrap.ASMainFelix$1.run(ASMainFelix.java:91)
        at java.lang.Thread.run(Thread.java:637)

"Low Memory Detector" daemon prio=5 tid=0x0000000101844800
nid=0x131983000 runnable [0x0000000000000000..0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=9 tid=0x0000000101843800 nid=0x131880000
waiting on condition [0x0000000000000000..0x000000013187e3e0]
    java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=9 tid=0x0000000101841800 nid=0x13177d000
waiting on condition [0x0000000000000000..0x000000013177b3e0]
    java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=9 tid=0x0000000101841000 nid=0x13167a000
runnable [0x0000000000000000..0x00000001316797d0]
    java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x0000000101829800 nid=0x131560000 in
Object.wait() [0x000000013155f000..0x000000013155fad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000010fac9f10> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x000000010fac9f10> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000101828800
nid=0x13145d000 in Object.wait() [0x000000013145c000..0x000000013145cad0]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000010fac9b88> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x000000010fac9b88> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x0000000101801000 nid=0x100401000 in Object.wait()
[0x0000000100400000..0x0000000100400c40]
    java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000010fb17bf0> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1161)
        - locked <0x000000010fb17bf0> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1214)
        at
com.sun.enterprise.glassfish.bootstrap.ASMainFelix.launchOSGiFW(ASMainFelix.java:110)
        at
com.sun.enterprise.glassfish.bootstrap.ASMainOSGi.run(ASMainOSGi.java:137)
        at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:65)

"VM Thread" prio=9 tid=0x0000000101820800 nid=0x13135a000 runnable

"VM Periodic Task Thread" prio=10 tid=0x0000000101846000 nid=0x131a86000
waiting on condition

"Exception Catcher Thread" prio=10 tid=0x0000000101802000
nid=0x100504000 runnable
JNI global references: 689


Found one Java-level deadlock:
=============================
"AutoDeployer":
   waiting to lock monitor 0x00000001338eee38 (object
0x000000010facca78, a java.util.logging.LogManager),
   which is held by "pool-3-thread-1"
"pool-3-thread-1":
   waiting to lock monitor 0x000000013214c448 (object
0x00000001252f1550, a java.lang.Class),
   which is held by "Thread-3"
"Thread-3":
   waiting to lock monitor 0x00000001338eee38 (object
0x000000010facca78, a java.util.logging.LogManager),
   which is held by "pool-3-thread-1"

Java stack information for the threads listed above:
===================================================
"AutoDeployer":
        at java.util.logging.LogManager.getLogger(LogManager.java:550)
        - waiting to lock <0x000000010facca78> (a java.util.logging.LogManager)
        at com.sun.logging.LogDomains.getLogger(LogDomains.java:246)
        at
org.glassfish.deployment.autodeploy.AutoDeployDirectoryScanner.<clinit>(AutoDeployDirectoryScanner.java:66)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.deployAll(AutoDeployer.java:385)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:343)
        at
org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:331)
        at
org.glassfish.deployment.autodeploy.AutoDeployService$1.run(AutoDeployService.java:200)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
"pool-3-thread-1":
        at java.util.logging.Logger.getLogger(Logger.java:274)
        - waiting to lock <0x00000001252f1550> (a java.lang.Class for
java.util.logging.Logger)
        at java.util.logging.LogManager.addLogger(LogManager.java:399)
        - locked <0x000000010facca78> (a java.util.logging.LogManager)
        at com.sun.logging.LogDomains.getLogger(LogDomains.java:331)
        at
com.sun.enterprise.util.i18n.StringManagerBase.<clinit>(StringManagerBase.java:101)
        at
com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.<clinit>(ResourceRecoveryManagerImpl.java:89)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at com.sun.hk2.component.ConstructorWomb.create(ConstructorWomb.java:72)
        at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:76)
        at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
        - locked <0x0000000110853138> (a com.sun.hk2.component.SingletonInhabitant)
        at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
        at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
        at
com.sun.enterprise.v3.server.AppServerStartup$2.run(AppServerStartup.java:197)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:637)
"Thread-3":
        at java.util.logging.LogManager.getLogger(LogManager.java:550)
        - waiting to lock <0x000000010facca78> (a java.util.logging.LogManager)
        at java.util.logging.Logger.getLogger(Logger.java:275)
        - locked <0x00000001252f1550> (a java.lang.Class for
java.util.logging.Logger)
        at com.sun.jmx.remote.util.ClassLogger.<init>(ClassLogger.java:37)
        at
javax.management.remote.JMXConnectorServerFactory.<clinit>(JMXConnectorServerFactory.java:182)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread$RMIConnectorStarter.startRMIConnector(ConnectorStartupService.java:282)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread.startConnector(ConnectorStartupService.java:309)
        at
org.glassfish.admin.mbeanserver.ConnectorStartupService$ConnectorsStarterThread.run(ConnectorStartupService.java:339)

Found 1 deadlock.