Thanks Mark this was helpful.Also Kedar suggested trying asadmin
start-domain --verbose domain1 and doing control break to get the thread
dump.
Here is the thread dump. Jan is looking into it
Regards,
Bhakti
Mark A. Basler wrote:
> HI Guys,
>
> I have used this before and it worked well. It is based on Code
> Injection.
>
> http://www.adaptj.com/root/main/stacktrace
>
> FYI...
>
> Hope this helps - Thanks - Mark
>
> Bhakti Mehta wrote:
>> Jan,
>> jstack is not present in windows bits of JDK and this bug is occuring
>> on Win. Is there something else we can use to find the deadlocks?
>>
>> Thanks,
>> Bhakti
>>
>> Jan Luehe wrote:
>>> Bhakti,
>>>
>>> Bhakti Mehta wrote On 02/27/07 12:44 PM,:
>>>
>>>> Jan,
>>>> I am not getting GF to startup as yet. It keeps hanging and is
>>>> stuck at this point. I did a fresh checkout and bootstrap-all and
>>>> build and configure-runtime
>>>
>>>
>>> hmm, this is weird, since none of the Tinderbox runs seem to have
>>> this problem.
>>>
>>> Can you send the output of jstack for your appserver process*?**
>>> *This may help us identify if/where a deadlock may be occurring.
>>>
>>> Thanks,
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>> Please let me know if I am missing something
>>>> Regards,
>>>> Bhakti
>>>> [#|2007-02-27T12:26:21.687-0800|INFO|sun-appserver9.1|javax.enterprise.system.to
>>>>
>>>> ols.deployment|_ThreadID=10;_ThreadName=main;|[AutoDeploy]
>>>> Selecting file C:\pub
>>>> lish\glassfish\lib\install\applications\__JWSappclients.ear for
>>>> autodeployment.|
>>>> #]^M
>>>> ^M
>>>> [#|2007-02-27T12:26:22.281-0800|INFO|sun-appserver9.1|javax.enterprise.system.to
>>>>
>>>> ols.deployment|_ThreadID=10;_ThreadName=main;|deployed with
>>>> moduleid = __JWSappc
>>>> lients|#]^M
>>>> ^M
>>>> [#|2007-02-27T12:26:22.343-0800|INFO|sun-appserver9.1|javax.enterprise.system.to
>>>>
>>>> ols.deployment|_ThreadID=10;_ThreadName=main;|[AutoDeploy]
>>>> Successfully autodepl
>>>> oyed :
>>>> C:\publish\glassfish\lib\install\applications\__JWSappclients.ear.|#]^M
>>>>
>>>> ^M
>>>> [#|2007-02-27T12:26:22.656-0800|INFO|sun-appserver9.1|javax.enterprise.system.co
>>>>
>>>> ntainer.web|_ThreadID=10;_ThreadName=main;|WEB0302: Starting
>>>> Sun-Java-System/App
>>>> lication-Server.|#]^M
>>>> ^M
>>>> [#|2007-02-27T12:26:27.578-0800|INFO|sun-appserver9.1|javax.enterprise.resource.
>>>>
>>>> webcontainer.jsf.config|_ThreadID=12;_ThreadName=pool-1-thread-7;;|Initializing
>>>>
>>>> Sun's JavaServer Faces implementation (1.2_03-b09-FCS) for context
>>>> ''|#]
>>>>
>>>> Jan Luehe wrote:
>>>>
>>>>> Hi Bhakti,
>>>>>
>>>>> Bhakti Mehta wrote On 02/26/07 03:20 PM,:
>>>>>
>>>>>> Hi,
>>>>>> Has anyone noticed similar problems?
>>>>>
>>>>>
>>>>>
>>>>> This is now fixed.
>>>>>
>>>>> Please update your ws with these files:
>>>>>
>>>>> /cvs/glassfish/appserv-core/src/java/com/sun/enterprise/web/EmbeddedWebContainer.java
>>>>>
>>>>> /cvs/glassfish/appserv-core/src/java/com/sun/enterprise/web/WebContainer.java
>>>>>
>>>>>
>>>>> Following is an explanation of what caused the
>>>>> java.io.FileNotFoundException:
>>>>>
>>>>> During startup, every webapp creates its own work directory:
>>>>>
>>>>> ${com.sun.aas.instanceRoot}/generated/jsp/j2ee-modules/<module-name>
>>>>>
>>>>> All of the parent directories of a webapp's work directory are shared
>>>>> across all webapps.
>>>>>
>>>>> While some of the shared parent directories, such as
>>>>>
>>>>> ${com.sun.aas.instanceRoot}
>>>>>
>>>>> already exist before startup, others, such as
>>>>>
>>>>> ${com.sun.aas.instanceRoot}/generated/jsp/j2ee-modules
>>>>>
>>>>> and
>>>>>
>>>>> ${com.sun.aas.instanceRoot}/generated/jsp/j2ee-apps
>>>>>
>>>>> are created during the startup of a webapp (unless they already
>>>>> exist),
>>>>> using java.io.File.mkdirs().
>>>>>
>>>>> Now that webapps are started concurrently, we've started seeing
>>>>> concurrency issues around java.io.File.mkdirs(), preventing some
>>>>> webapps from creating their own work directory and "tldCache.ser"
>>>>> resource in it, causing the java.io.FileNotFoundException you've
>>>>> seen.
>>>>>
>>>>> The fix is to create any shared parent directories before starting
>>>>> the
>>>>> webapps concurrently, thus avoiding any concurrency issues around
>>>>> .java.io.File.mkdirs() without adding the need for synchronization.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jan
>>>>>
>>>>>> Regards,
>>>>>> Bhakti
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> Subject:
>>>>>> Startup errors related to FileNotFoundException tlcCache.ser
>>>>>> From:
>>>>>> Bhakti Mehta <Bhakti.Mehta_at_sun.com>
>>>>>> Date:
>>>>>> Mon, 26 Feb 2007 15:13:17 -0800
>>>>>> To:
>>>>>> Vijay Ramachandran <Vijay.Ramachandran_at_Sun.COM>,
>>>>>> Mike.Grogan_at_Sun.COM, Dinesh Patil <Dinesh.Patil_at_Sun.COM>
>>>>>>
>>>>>> To:
>>>>>> Vijay Ramachandran <Vijay.Ramachandran_at_Sun.COM>,
>>>>>> Mike.Grogan_at_Sun.COM, Dinesh Patil <Dinesh.Patil_at_Sun.COM>
>>>>>>
>>>>>>
>>>>>> Hi all,
>>>>>> I did maven checkout
>>>>>> maven bootstrap-all build
>>>>>> maven configure-runtime
>>>>>> On starting GF I get this error related to tldCache.ser and
>>>>>> scratchDir is unusable?
>>>>>> Please can you suggest?
>>>>>> Regards,
>>>>>> Bhakti
>>>>>>
>>>>>> [#|2007-02-26T15:09:19.500-0800|WARNING|sun-appserver9.1|javax.enterprise.syste
>>>>>>
>>>>>> .stream.err|_ThreadID=12;_ThreadName=pool-1-thread-6;_RequestID=dfe77e63-9aaf-4
>>>>>>
>>>>>> 49-8751-fe3c48cb7ffe;|java.io.FileNotFoundException:
>>>>>> C:\publish\glassfish\domai
>>>>>> s\domain1\generated\jsp\j2ee-modules\__default-web-module-server\tldCache.ser
>>>>>> (
>>>>>> he system cannot find the path specified)
>>>>>> at java.io.FileOutputStream.open(Native Method)
>>>>>> at
>>>>>> java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>>>>>> at
>>>>>> java.io.FileOutputStream.<init>(FileOutputStream.java:131)
>>>>>> at
>>>>>> org.apache.catalina.startup.TldConfig.execute(TldConfig.java:426)
>>>>>> at
>>>>>> org.apache.catalina.core.StandardContext.start(StandardContext.java:
>>>>>> 053)
>>>>>> at
>>>>>> com.sun.enterprise.web.WebModule.start(WebModule.java:304)
>>>>>> at
>>>>>> com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:
>>>>>> 5)
>>>>>> at
>>>>>> com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBa
>>>>>> e.java:283)
>>>>>> at
>>>>>> com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.j
>>>>>> va:317)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>> 7)
>>>>>> at
>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe
>>>>>> utor.java:650)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>> .java:675)
>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>> |#]
>>>>>>
>>>>>> [#|2007-02-26T15:09:19.546-0800|WARNING|sun-appserver9.1|javax.enterprise.syste
>>>>>>
>>>>>> .stream.err|_ThreadID=13;_ThreadName=pool-1-thread-3;_RequestID=9d49f734-5260-4
>>>>>>
>>>>>> 71-a6da-0856a5beb2f1;|
>>>>>> java.io.FileNotFoundException:
>>>>>> C:\publish\glassfish\domains\domain1\generated\j
>>>>>> p\j2ee-modules\adminapp\tldCache.ser (The system cannot find the
>>>>>> path specified
>>>>>>
>>>>>> at java.io.FileOutputStream.open(Native Method)
>>>>>> at
>>>>>> java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>>>>>> at
>>>>>> java.io.FileOutputStream.<init>(FileOutputStream.java:131)
>>>>>> at
>>>>>> org.apache.catalina.startup.TldConfig.execute(TldConfig.java:426)
>>>>>> at
>>>>>> org.apache.catalina.core.StandardContext.start(StandardContext.java:
>>>>>> 053)
>>>>>> at
>>>>>> com.sun.enterprise.web.WebModule.start(WebModule.java:304)
>>>>>> at
>>>>>> com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:
>>>>>> 5)
>>>>>> at
>>>>>> com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBa
>>>>>> e.java:283)
>>>>>> at
>>>>>> com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.j
>>>>>> va:317)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>> 7)
>>>>>> at
>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe
>>>>>> utor.java:650)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>> .java:675)
>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>> |#]
>>>>>>
>>>>>> [#|2007-02-26T15:09:19.625-0800|SEVERE|sun-appserver9.1|org.apache.jasper.Embed
>>>>>>
>>>>>> edServletOptions|_ThreadID=12;_ThreadName=pool-1-thread-6;_RequestID=dfe77e63-9
>>>>>>
>>>>>> af-4449-8751-fe3c48cb7ffe;|PWC5963: The scratchDir you specified:
>>>>>> C:\publish\gl
>>>>>> ssfish\domains\domain1\generated\jsp\j2ee-modules\__default-web-module-server
>>>>>> i
>>>>>> unusable.|#]
>>>>>>
>>>>>> [#|2007-02-26T15:09:19.625-0800|SEVERE|sun-appserver9.1|org.apache.jasper.Embed
>>>>>>
>>>>>> edServletOptions|_ThreadID=13;_ThreadName=pool-1-thread-3;_RequestID=9d49f734-5
>>>>>>
>>>>>> 60-4371-a6da-0856a5beb2f1;|PWC5963: The scratchDir you specified:
>>>>>> C:\publish\gl
>>>>>> ssfish\domains\domain1\generated\jsp\j2ee-modules\adminapp is
>>>>>> unusable.|#]
>>>>>>
>>>>>> [#|2007-02-26T15:09:20.281-0800|INFO|sun-appserver9.1|javax.enterprise.resource
>>>>>>
>>>>>> webcontainer.jsf.config|_ThreadID=14;_ThreadName=pool-1-thread-7;;|Initializing
>>>>>>
>>>>>> Sun's JavaServer Faces implementation (1.2_03-b09-FCS) for
>>>>>> context ''|#]
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>
Full thread dump Java HotSpot(TM) Client VM (1.5.0_10-b03 mixed mode):
"StackTrace Remote Thread" prio=6 tid=0x26f3ecf0 nid=0x1708 waiting on condition [0x00000000..0x2840fb6c]
"RMI ConnectionExpiration-[129.145.133.92:1126]" daemon prio=6 tid=0x2778c138 nid=0x1f4 waiting on condition [0x282bf000..0x282bfae8]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Connection(6)-129.145.133.92" daemon prio=6 tid=0x27612df8 nid=0x680 runnable [0x281bf000..0x281bfb68]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x031285a0> (a sun.nio.ch.Util$1)
- locked <0x03128590> (a java.util.Collections$UnmodifiableSet)
- locked <0x03128370> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:120)
at com.sun.enterprise.server.ss.provider.ASInputStream.waitForSelect(ASInputStream.java:119)
at com.sun.enterprise.server.ss.provider.ASInputStream.read(ASInputStream.java:99)
at com.sun.enterprise.server.ss.provider.ASInputStream.read(ASInputStream.java:91)
- locked <0x03128328> (a com.sun.enterprise.server.ss.provider.ASInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
- locked <0x0312c670> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
"RMI ConnectionExpiration-[129.145.133.92:1097,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_a]" daemon prio=6 tid=0x272239a0 nid=0x1598 waiting on condition [0x2817f000..0x2817fbe8]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Connection(5)-127.0.0.1" daemon prio=6 tid=0x26fb5e08 nid=0x1594 runnable [0x27fff000..0x27fffc68]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x030d5aa8> (a sun.nio.ch.Util$1)
- locked <0x030d5a98> (a java.util.Collections$UnmodifiableSet)
- locked <0x030d5878> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:120)
at com.sun.enterprise.server.ss.provider.ASInputStream.waitForSelect(ASInputStream.java:119)
at com.sun.enterprise.server.ss.provider.ASInputStream.read(ASInputStream.java:99)
at com.sun.enterprise.server.ss.provider.ASInputStream.read(ASInputStream.java:91)
- locked <0x030d5830> (a com.sun.enterprise.server.ss.provider.ASInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
- locked <0x030d9b78> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]" daemon prio=6 tid=0x27705990 nid=0xb48 waiting on condition [0x286bf000..0x286bfce8]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1795)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[]]" daemon prio=6 tid=0x26ec0fd0 nid=0x890 waiting on condition [0x2867f000..0x2867fd68]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1795)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon prio=6 tid=0x276d4ca8 nid=0x1784 waiting on condition [0x2863f000..0x2863f9e8]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1795)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]" daemon prio=6 tid=0x276d4e30 nid=0x1788 waiting on condition [0x285ff000..0x285ffae8]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1795)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[/web1]]" daemon prio=6 tid=0x276e2d00 nid=0x17e8 waiting on condition [0x285bf000..0x285bfa68]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1795)
at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-7" prio=6 tid=0x270fc368 nid=0x17d4 waiting on condition [0x2857f000..0x2857fb68]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:681)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:809)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1124)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:423)
at org.apache.catalina.core.ContainerBase.getLogger(ContainerBase.java:455)
at org.apache.catalina.core.ContainerBase.getLogger(ContainerBase.java:463)
at org.apache.catalina.core.ApplicationContext.log(ApplicationContext.java:805)
at org.apache.catalina.core.ApplicationContextFacade.log(ApplicationContextFacade.java:294)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4340)
- locked <0x0dd93f00> (a java.util.HashMap)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5125)
- locked <0x0dd88e80> (a com.sun.enterprise.web.WebModule)
at com.sun.enterprise.web.WebModule.start(WebModule.java:304)
- locked <0x0dd88e80> (a com.sun.enterprise.web.WebModule)
at com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:45)
at com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBase.java:283)
at com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.java:317)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
"RMI RenewClean-[129.145.133.92:1126]" daemon prio=6 tid=0x26eedd20 nid=0x1794 in Object.wait() [0x2837f000..0x2837fb68]
at java.lang.Object.wait(Native Method)
- waiting on <0x0dc66020> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x0dc66020> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Accept-0" daemon prio=6 tid=0x2703b250 nid=0x177c runnable [0x27ebf000..0x27ebfce8]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0x0dc6b5c0> (a java.lang.Object)
at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
- locked <0x0dc6b5c8> (a java.lang.Object)
at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:174)
at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:154)
- locked <0x0dc6b5f0> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:161)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Accept-8686" daemon prio=6 tid=0x26ec2bd8 nid=0x1780 runnable [0x2833f000..0x2833fd68]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0x0d6dfb98> (a java.lang.Object)
at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
- locked <0x0d6dfb78> (a java.lang.Object)
at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:174)
at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:154)
- locked <0x0d6deac0> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:161)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)
"Thread-14" prio=6 tid=0x26ff7978 nid=0x17bc waiting on condition [0x2827f000..0x2827fae8]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
at com.sun.enterprise.management.support.LoaderRegThread.process(LoaderRegThread.java:232)
at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:143)
"Thread-13" prio=6 tid=0x26ff42d8 nid=0x17ac waiting on condition [0x2823f000..0x2823fb68]
at java.lang.Thread.sleep(Native Method)
at com.sun.enterprise.management.support.LoaderBase.mySleep(LoaderBase.java:223)
at com.sun.enterprise.management.support.Loader$DeferredRegistrationThread.run(Loader.java:376)
"Thread-12" prio=6 tid=0x27027a10 nid=0x17c8 waiting on condition [0x281ff000..0x281ffbe8]
at java.lang.Thread.sleep(Native Method)
at com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:141)
at java.lang.Thread.run(Thread.java:595)
"RMI RenewClean-[129.145.133.92:1097,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_a]" daemon prio=6 tid=0x27034480 nid=0x1710 in Object.wait() [0x280ff000..0x280ffd68]
at java.lang.Object.wait(Native Method)
- waiting on <0x0da92fb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x0da92fb8> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
at java.lang.Thread.run(Thread.java:595)
"Timer-4" prio=6 tid=0x26fcef78 nid=0x1718 in Object.wait() [0x280bf000..0x280bf9e8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0da930a0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x0da930a0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer-3" daemon prio=6 tid=0x26fd2e10 nid=0x1728 in Object.wait() [0x2807f000..0x2807fa68]
at java.lang.Object.wait(Native Method)
- waiting on <0x0da6dc10> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x0da6dc10> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer-2" prio=6 tid=0x27672fe8 nid=0x1730 in Object.wait() [0x2803f000..0x2803fae8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0da2a7b8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x0da2a7b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer-1" prio=6 tid=0x27390bf8 nid=0x1738 in Object.wait() [0x27fbf000..0x27fbfbe8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d99e9f8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x0d99e9f8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"RMI LeaseChecker" daemon prio=6 tid=0x27687d68 nid=0x1734 waiting on condition [0x27f7f000..0x27f7fc68]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
at java.lang.Thread.run(Thread.java:595)
"Thread-7" prio=6 tid=0x27652ea0 nid=0x12ac runnable [0x27eff000..0x27effd68]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0d6c4e78> (a sun.nio.ch.Util$1)
- locked <0x0d6c4e88> (a java.util.Collections$UnmodifiableSet)
- locked <0x0d6c4e00> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:128)
at com.sun.enterprise.server.ss.ASSocketService$EntryPointThread.run(ASSocketService.java:492)
"Thread-5" prio=6 tid=0x2764a548 nid=0x171c runnable [0x27e7f000..0x27e7fa68]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0d6cb100> (a sun.nio.ch.Util$1)
- locked <0x0d6cb110> (a java.util.Collections$UnmodifiableSet)
- locked <0x0d6cb088> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:128)
at com.sun.enterprise.server.ss.ASSocketService$EntryPointThread.run(ASSocketService.java:492)
"Thread-4" prio=6 tid=0x27612148 nid=0x1714 runnable [0x27e3f000..0x27e3fae8]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0d6ce278> (a sun.nio.ch.Util$1)
- locked <0x0d6ce288> (a java.util.Collections$UnmodifiableSet)
- locked <0x0d6ce200> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:128)
at com.sun.enterprise.server.ss.ASSocketService$EntryPointThread.run(ASSocketService.java:492)
"Thread-3" prio=6 tid=0x2764cf30 nid=0x1700 runnable [0x27dff000..0x27dffb68]
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:275)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:257)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:138)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0d6d13f0> (a sun.nio.ch.Util$1)
- locked <0x0d6d1400> (a java.util.Collections$UnmodifiableSet)
- locked <0x0d6d1378> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:128)
at com.sun.enterprise.server.ss.ASSocketService$EntryPointThread.run(ASSocketService.java:492)
"GC Daemon" daemon prio=2 tid=0x27605cc0 nid=0x16f8 in Object.wait() [0x27dbf000..0x27dbfbe8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d6b2b38> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x0d6b2b38> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=6 tid=0x27607ef0 nid=0x12e4 in Object.wait() [0x27d7f000..0x27d7fc68]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d6b2ba0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x0d6b2ba0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
at java.lang.Thread.run(Thread.java:595)
"Timer-0" daemon prio=6 tid=0x275ddda8 nid=0x16f0 in Object.wait() [0x27d3f000..0x27d3fce8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d6b2c30> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x0d6b2c30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-0" daemon prio=6 tid=0x27614bb8 nid=0x1200 runnable [0x27cff000..0x27cffd68]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0x0d6b2d10> (a java.lang.Object)
at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
- locked <0x0d6b2d18> (a java.lang.Object)
at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:174)
at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:154)
- locked <0x0d6b2d40> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:161)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-1" prio=2 tid=0x27148cc8 nid=0x16cc waiting on condition [0x27c7f000..0x27c7f9e8]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:681)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:837)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1144)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
at com.sun.appserv.management.util.misc.RunnableBase.waitDone(RunnableBase.java:367)
at com.sun.enterprise.web.VirtualServer.startChildren(VirtualServer.java:1680)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1220)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:938)
- locked <0x0d9d8208> (a com.sun.enterprise.web.VirtualServer)
at com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:45)
at com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBase.java:283)
at com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.java:317)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
"Low Memory Detector" daemon prio=6 tid=0x0096f3c0 nid=0x350 runnable [0x00000000..0x00000000]
"CompilerThread0" daemon prio=10 tid=0x0096dfc0 nid=0x16b4 waiting on condition [0x00000000..0x26d3f848]
"Signal Dispatcher" daemon prio=10 tid=0x0096d3d0 nid=0x133c runnable [0x00000000..0x00000000]
"Finalizer" daemon prio=8 tid=0x009641a0 nid=0x1348 in Object.wait() [0x26cbf000..0x26cbfc68]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d4df410> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x0d4df410> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00962d30 nid=0x1368 in Object.wait() [0x26c7f000..0x26c7fce8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0d4df490> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x0d4df490> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x0003c098 nid=0x1328 waiting on condition [0x0007f000..0x0007fc3c]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:681)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:837)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1144)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
at com.sun.appserv.management.util.misc.RunnableBase.waitDone(RunnableBase.java:367)
at com.sun.enterprise.web.EmbeddedWebContainer$WebEngine.startChildren(EmbeddedWebContainer.java:447)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1220)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:520)
at org.apache.catalina.startup.Embedded.start(Embedded.java:916)
at com.sun.enterprise.web.WebContainer.start(WebContainer.java:853)
at com.sun.enterprise.web.PEWebContainer.startInstance(PEWebContainer.java:744)
at com.sun.enterprise.web.PEWebContainerLifecycle.onStartup(PEWebContainerLifecycle.java:71)
at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:332)
at com.sun.enterprise.server.ondemand.OnDemandServer.onStartup(OnDemandServer.java:112)
at com.sun.enterprise.server.PEMain.run(PEMain.java:379)
at com.sun.enterprise.server.PEMain.main(PEMain.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.enterprise.server.PELaunch.main(PELaunch.java:274)
"VM Thread" prio=10 tid=0x00923268 nid=0x1208 runnable
"VM Periodic Task Thread" prio=10 tid=0x00970618 nid=0x354 waiting on condition