dev@glassfish.java.net

webservice hang

From: Harsha R A <Harsha.Ra_at_Sun.COM>
Date: Fri, 27 Apr 2007 22:28:49 +0530

Hi,
I am trying to invoke a webservice on glassfish v2 and I find the server
hanging at the first request. I think this is very similar to the bug

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6382475

. Attached is the jstack output when the hang happens. The hang because
it causes a infinity request loop I think. As described in the above
bug, which was for AS 9.0

"When there are imported WSDLs and schema in the service's main wsdl, we
replace them with our own __context$publish URI whose context root is
under our control. But the servlet that servies this context root has
not yet completed its init. Hence there is a infinite loop kind of
situation. The fix will be to do jaxws.init upon the first doPost - this
will cause the service request to be slow"

The fix for this can be seen here

https://glassfish.dev.java.net/source/browse/glassfish/appserv-core/src/java/com/sun/enterprise/webservice/JAXWSServlet.java?rev=1.24&only_with_tag=SJSAS-9_0-B37-RC-15_Feb_2006&view=log


I think this fix needs to be ported to glassfish v2 too.


Thanks
Harsha




2007-04-27 18:34:12
Full thread dump Java HotSpot(TM) Server VM (1.6.0_01-b06 mixed mode):

"Attach Listener" daemon prio=10 tid=0x014ba000 nid=0x6b waiting on condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/wsi-integrated]]" daemon prio=10 tid=0x011f2400 nid=0x6a waiting on condition [0xdcbff000..0xdcbff8f0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"Timer-7" daemon prio=10 tid=0x00dc2400 nid=0x65 in Object.wait() [0xdec7f000..0xdec7fa70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe70132a0> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0xe70132a0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"httpSSLWorkerThread-8080-3" daemon prio=10 tid=0x010fa000 nid=0x64 runnable [0xded7d000..0xded7fbf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        - locked <0xe1610678> (a java.io.BufferedInputStream)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:647)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:592)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
        - locked <0xe1606cd8> (a sun.net.www.protocol.http.HttpURLConnection)
        at java.net.URL.openStream(URL.java:1009)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:785)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:236)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:107)
        at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:226)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:189)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:159)
        at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:81)
        at javax.xml.ws.Service.<init>(Service.java:56)
        at com.sun.wsi.scm.warehouseb.client.WarehouseBService.<init>(WarehouseBService.java:37)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.sun.enterprise.naming.NamingManagerImpl.initiateInstance(NamingManagerImpl.java:1074)
        at com.sun.enterprise.naming.NamingManagerImpl.getClientServiceObject(NamingManagerImpl.java:1117)
        at com.sun.enterprise.naming.NamingManagerImpl.lookup(NamingManagerImpl.java:979)
        at com.sun.enterprise.naming.java.javaURLContext.lookup(javaURLContext.java:160)
        at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:324)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at com.sun.enterprise.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:274)
        at com.sun.enterprise.util.InjectionManagerImpl.inject(InjectionManagerImpl.java:193)
        at com.sun.enterprise.util.InjectionManagerImpl.injectInstance(InjectionManagerImpl.java:104)
        at com.sun.web.server.J2EEInstanceListener.handleBeforeEvent(J2EEInstanceListener.java:246)
        at com.sun.web.server.J2EEInstanceListener.instanceEvent(J2EEInstanceListener.java:106)
        at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:318)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1150)
        - locked <0xe0c20368> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:819)
        - locked <0xe0c20368> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:184)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:346)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)

"httpSSLWorkerThread-8080-2" daemon prio=10 tid=0x00a91400 nid=0x63 runnable [0xdef7d000..0xdef7fb70]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        - locked <0xe12aaf20> (a java.io.BufferedInputStream)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:647)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:592)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
        - locked <0xe12a1460> (a sun.net.www.protocol.http.HttpURLConnection)
        at java.net.URL.openStream(URL.java:1009)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:785)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:236)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:107)
        at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:226)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:189)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:159)
        at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:81)
        at javax.xml.ws.Service.<init>(Service.java:56)
        at com.sun.wsi.scm.warehousec.client.WarehouseCService.<init>(WarehouseCService.java:37)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.sun.enterprise.naming.NamingManagerImpl.initiateInstance(NamingManagerImpl.java:1074)
        at com.sun.enterprise.naming.NamingManagerImpl.getClientServiceObject(NamingManagerImpl.java:1117)
        at com.sun.enterprise.naming.NamingManagerImpl.lookup(NamingManagerImpl.java:979)
        at com.sun.enterprise.naming.java.javaURLContext.lookup(javaURLContext.java:160)
        at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:324)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at com.sun.enterprise.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:274)
        at com.sun.enterprise.util.InjectionManagerImpl.inject(InjectionManagerImpl.java:193)
        at com.sun.enterprise.util.InjectionManagerImpl.injectInstance(InjectionManagerImpl.java:104)
        at com.sun.web.server.J2EEInstanceListener.handleBeforeEvent(J2EEInstanceListener.java:246)
        at com.sun.web.server.J2EEInstanceListener.instanceEvent(J2EEInstanceListener.java:106)
        at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:318)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1150)
        - locked <0xe0c1abc8> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:819)
        - locked <0xe0c1abc8> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:184)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:346)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)

"Thread-29" daemon prio=10 tid=0x008af000 nid=0x5f waiting on condition [0xde67f000..0xde67f970]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.enterprise.admin.jmx.remote.server.notification.ServerNotificationManager.run(ServerNotificationManager.java:215)
        at java.lang.Thread.run(Thread.java:619)

"Timer-6" daemon prio=10 tid=0x00e5a800 nid=0x5e in Object.wait() [0xda5ff000..0xda5ffaf0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe6a416d8> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0xe6a416d8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-5" daemon prio=10 tid=0x0160d800 nid=0x5d in Object.wait() [0xda6ff000..0xda6ffa70]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe69245e8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xe69245e8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-4" prio=10 tid=0x01572400 nid=0x5c in Object.wait() [0xda7ff000..0xda7ffbf0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe67e3dd8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xe67e3dd8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"DestroyJavaVM" prio=10 tid=0x00037000 nid=0x2 waiting on condition [0x00000000..0xfe67fb58]
   java.lang.Thread.State: RUNNABLE

"Timer-3" daemon prio=10 tid=0x01571000 nid=0x5b in Object.wait() [0xda8ff000..0xda8ffb70]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe67e3e20> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xe67e3e20> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x00b22000 nid=0x59 waiting on condition [0xda9ff000..0xda9ff870]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[/web1]]" daemon prio=10 tid=0x00cbb000 nid=0x57 waiting on condition [0xdabff000..0xdabff970]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/WebApplication8]]" daemon prio=10 tid=0x00bbcc00 nid=0x56 waiting on condition [0xdacff000..0xdacffaf0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[]]" daemon prio=10 tid=0x015f0000 nid=0x55 waiting on condition [0xdadff000..0xdadffa70]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon prio=10 tid=0x00f9b800 nid=0x54 waiting on condition [0xdaeff000..0xdaeffbf0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/WebApplication11]]" daemon prio=10 tid=0x00e60c00 nid=0x53 waiting on condition [0xdafff000..0xdafffb70]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/WebApplication12]]" daemon prio=10 tid=0x00e60800 nid=0x52 waiting on condition [0xdccff000..0xdccff8f0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1787)
        at java.lang.Thread.run(Thread.java:619)

"httpSSLWorkerThread-8181-1" daemon prio=10 tid=0x00b4a400 nid=0x51 in Object.wait() [0xdb0ff000..0xdb0ff870]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe654bd30> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:278)
        - locked <0xe654bd30> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:91)

"httpWorkerThread-4848-1" daemon prio=10 tid=0x00b49800 nid=0x50 in Object.wait() [0xdb1ff000..0xdb1ff970]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe654bd80> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:278)
        - locked <0xe654bd80> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:101)

"httpWorkerThread-4848-0" daemon prio=10 tid=0x00b49400 nid=0x4f in Object.wait() [0xdb2ff000..0xdb2ff9f0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe654bd80> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:278)
        - locked <0xe654bd80> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:101)

"httpSSLWorkerThread-8181-0" daemon prio=10 tid=0x00b48c00 nid=0x4e in Object.wait() [0xdb3ff000..0xdb3ffaf0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe654bd30> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:278)
        - locked <0xe654bd30> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:91)

"SelectorThread-4848" prio=10 tid=0x00f55400 nid=0x4d runnable [0xdb4ff000..0xdb4ffa70]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0xe654be50> (a sun.nio.ch.Util$1)
        - locked <0xe654be60> (a java.util.Collections$UnmodifiableSet)
        - locked <0xe654be08> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1258)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1205)
        - locked <0xe654be70> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1168)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1144)

"SelectorThread-8181" prio=10 tid=0x00f1fc00 nid=0x4c runnable [0xdb5ff000..0xdb5ffbf0]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0xe654bf00> (a sun.nio.ch.Util$1)
        - locked <0xe654bf10> (a java.util.Collections$UnmodifiableSet)
        - locked <0xe654beb8> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1258)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1205)
        - locked <0xe636b620> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1168)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1144)

"httpSSLWorkerThread-8080-1" daemon prio=10 tid=0x00f89c00 nid=0x4b runnable [0xdb6fc000..0xdb6ff8f0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        - locked <0xe15b4cd0> (a java.io.BufferedInputStream)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:647)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:592)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
        - locked <0xe15ab330> (a sun.net.www.protocol.http.HttpURLConnection)
        at java.net.URL.openStream(URL.java:1009)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:785)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:236)
        at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:107)
        at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:226)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:189)
        at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:159)
        at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:81)
        at javax.xml.ws.Service.<init>(Service.java:56)
        at com.sun.wsi.scm.warehouseb.client.WarehouseBService.<init>(WarehouseBService.java:37)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.sun.enterprise.naming.NamingManagerImpl.initiateInstance(NamingManagerImpl.java:1074)
        at com.sun.enterprise.naming.NamingManagerImpl.getClientServiceObject(NamingManagerImpl.java:1117)
        at com.sun.enterprise.naming.NamingManagerImpl.lookup(NamingManagerImpl.java:979)
        at com.sun.enterprise.naming.java.javaURLContext.lookup(javaURLContext.java:160)
        at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:324)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at com.sun.enterprise.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:274)
        at com.sun.enterprise.util.InjectionManagerImpl.inject(InjectionManagerImpl.java:193)
        at com.sun.enterprise.util.InjectionManagerImpl.injectInstance(InjectionManagerImpl.java:104)
        at com.sun.web.server.J2EEInstanceListener.handleBeforeEvent(J2EEInstanceListener.java:246)
        at com.sun.web.server.J2EEInstanceListener.instanceEvent(J2EEInstanceListener.java:106)
        at org.apache.catalina.util.InstanceSupport.fireInstanceEvent(InstanceSupport.java:318)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1150)
        - locked <0xe0c1cf60> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:819)
        - locked <0xe0c1cf60> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:184)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:346)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)

"httpSSLWorkerThread-8080-0" daemon prio=10 tid=0x00f9d400 nid=0x4a waiting for monitor entry [0xdb7fe000..0xdb7ffb70]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:814)
        - waiting to lock <0xe0c20368> (a org.apache.catalina.core.StandardWrapper)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:184)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:346)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)

"SelectorThread-8080" prio=10 tid=0x009fb400 nid=0x49 runnable [0xdb8ff000..0xdb8ff870]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0xe636b718> (a sun.nio.ch.Util$1)
        - locked <0xe636b728> (a java.util.Collections$UnmodifiableSet)
        - locked <0xe636b6d0> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1258)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1205)
        - locked <0xe636b738> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1168)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1144)

"iMQReadChannel-0" daemon prio=10 tid=0x00493400 nid=0x48 runnable [0xdb9ff000..0xdb9ff9f0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        - locked <0xe60c52c8> (a java.io.BufferedInputStream)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:263)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:183)
        at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:143)
        at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:73)
        - locked <0xe60c5340> (a com.sun.messaging.jmq.io.ReadWritePacket)
        at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1730)
        at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1135)
        at java.lang.Thread.run(Thread.java:619)

"imqConnectionFlowControl-0" daemon prio=10 tid=0x00492c00 nid=0x47 in Object.wait() [0xdbaff000..0xdbaff970]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe60c5520> (a com.sun.messaging.jmq.jmsclient.FlowControl)
        at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:290)
        - locked <0xe60c5520> (a com.sun.messaging.jmq.jmsclient.FlowControl)
        at java.lang.Thread.run(Thread.java:619)

"Thread-jms[1]" prio=10 tid=0x00a64800 nid=0x46 in Object.wait() [0xdbbff000..0xdbbffa70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe60c5548> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.waitForWork(IMQIPConnection.java:1826)
        - locked <0xe60c5548> (a java.lang.Object)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.writeData(IMQIPConnection.java:1538)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:623)
        at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:159)
        at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:477)
        at java.lang.Thread.run(Thread.java:619)

"Thread-jms[0]" prio=10 tid=0x000cec00 nid=0x45 runnable [0xdbcff000..0xdbcffaf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        - locked <0xe60c5ef0> (a java.io.BufferedInputStream)
        at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1946)
        at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1914)
        at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1841)
        - locked <0xe121dee0> (a com.sun.messaging.jmq.io.Packet)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1337)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:636)
        at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:159)
        at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:477)
        at java.lang.Thread.run(Thread.java:619)

"ClusterServiceListener" daemon prio=10 tid=0x00a53000 nid=0x44 runnable [0xdbdff000..0xdbdffbf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe60c5f68> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.messaging.jmq.jmsserver.multibroker.fullyconnected.ClusterServiceListener.run(ClusterImpl.java:1742)

"MessageBusCallbackDispatcher" daemon prio=10 tid=0x00a65800 nid=0x43 in Object.wait() [0xdbeff000..0xdbeffb70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe60c6088> (a java.util.LinkedList)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.messaging.jmq.jmsserver.multibroker.CallbackDispatcher.run(CallbackDispatcher.java:316)
        - locked <0xe60c6088> (a java.util.LinkedList)

"jms_ACCEPT" prio=10 tid=0x00496400 nid=0x42 runnable [0xdbfff000..0xdbfff8f0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe6031be8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProtocol.java:270)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPService.java:565)
        at java.lang.Thread.run(Thread.java:619)

"admin_ACCEPT" prio=10 tid=0x00a7f000 nid=0x41 runnable [0xdc0ff000..0xdc0ff870]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe6032720> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProtocol.java:270)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPService.java:565)
        at java.lang.Thread.run(Thread.java:619)

"RMI RenewClean-[129.158.229.66:56227]" daemon prio=10 tid=0x00a6cc00 nid=0x3f in Object.wait() [0xdc2ff000..0xdc2ff970]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe602d720> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xe602d720> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x010ad000 nid=0x3e runnable [0xdc3ff000..0xdc3ffaf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe602c878> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"ClusterDiscoveryService" daemon prio=10 tid=0x01080400 nid=0x3d runnable [0xdc4ff000..0xdc4ffa70]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe6014fe8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.messaging.jmq.jmsserver.service.ClusterDiscoveryService.run(ClusterDiscoveryService.java:305)
        at java.lang.Thread.run(Thread.java:619)

"Broker Monitor" daemon prio=10 tid=0x00a7b400 nid=0x3c in Object.wait() [0xdc5ff000..0xdc5ffbf0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe600f338> (a java.util.Collections$SynchronizedSet)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.messaging.jmq.jmsserver.core.cluster.BrokerConsumers.run(MultibrokerRouter.java:1192)
        - locked <0xe600f338> (a java.util.Collections$SynchronizedSet)
        at java.lang.Thread.run(Thread.java:619)

"JMQPortMapper" daemon prio=10 tid=0x00a80400 nid=0x3b runnable [0xdc9ff000..0xdc9ffb70]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe60350b0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.messaging.jmq.jmsserver.service.PortMapper.run(PortMapper.java:464)
        at java.lang.Thread.run(Thread.java:619)

"JMQTimerThread" daemon prio=10 tid=0x000d5400 nid=0x3a in Object.wait() [0xdcaff000..0xdcaff8f0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe6035210> (a com.sun.messaging.jmq.util.timer.TaskQueue)
        at com.sun.messaging.jmq.util.timer.TimerThread.mainLoop(JMQTimer.java:494)
        - locked <0xe6035210> (a com.sun.messaging.jmq.util.timer.TaskQueue)
        at com.sun.messaging.jmq.util.timer.TimerThread.run(JMQTimer.java:428)

"RMI RenewClean-[129.158.229.66:56221]" daemon prio=10 tid=0x005d6000 nid=0x32 in Object.wait() [0xdd1ff000..0xdd1ff8f0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5f76cf0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xe5f76cf0> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x005da800 nid=0x2f runnable [0xdcfff000..0xdcfff970]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe5f6ed80> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-8686" daemon prio=10 tid=0x010b2800 nid=0x2e runnable [0xdd0ff000..0xdd0ffaf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe5f86ba8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Thread-11" prio=10 tid=0x00391400 nid=0x2b waiting on condition [0xdd3ff000..0xdd3ffb70]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0xe5dbcda8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at com.sun.enterprise.management.support.LoaderRegThread.process(LoaderRegThread.java:232)
        at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:143)

"Thread-10" prio=10 tid=0x00390000 nid=0x2a waiting on condition [0xdd4ff000..0xdd4ff8f0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.enterprise.management.support.LoaderBase.mySleep(LoaderBase.java:223)
        at com.sun.enterprise.management.support.Loader$DeferredRegistrationThread.run(Loader.java:376)

"Thread-9" prio=10 tid=0x0038a000 nid=0x29 waiting on condition [0xdd5ff000..0xdd5ff870]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:141)
        at java.lang.Thread.run(Thread.java:619)

"RMI RenewClean-[129.158.229.66:56215,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_]" daemon prio=10 tid=0x00388800 nid=0x27 in Object.wait() [0xdd7ff000..0xdd7ff970]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5dbce78> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xe5dbce78> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"Thread-7" prio=10 tid=0x003d0400 nid=0x25 in Object.wait() [0xdd8ff000..0xdd8ffa70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5d52598> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:804)
        - locked <0xe5d52598> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)

"p: thread-pool-1; w: 2" daemon prio=10 tid=0x007db000 nid=0x24 runnable [0xdd9ff000..0xdd9ffb70]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe5ceb608> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:221)
        at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:72)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:502)

"p: thread-pool-1; w: 1" daemon prio=10 tid=0x007dac00 nid=0x23 runnable [0xddaff000..0xddaffbf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe5d34ee8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:221)
        at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:72)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:502)

"SelectorThread" daemon prio=10 tid=0x00925400 nid=0x22 runnable [0xddbff000..0xddbff8f0]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0xe5ceb728> (a sun.nio.ch.Util$1)
        - locked <0xe5ceb738> (a java.util.Collections$UnmodifiableSet)
        - locked <0xe5ceb6e0> (a sun.nio.ch.DevPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:257)

"Timer-2" daemon prio=10 tid=0x01042800 nid=0x21 in Object.wait() [0xddcff000..0xddcff870]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5bfdc60> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0xe5bfdc60> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-1" prio=10 tid=0x00ab7000 nid=0x20 in Object.wait() [0xdddff000..0xdddff9f0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5bfdc70> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xe5bfdc70> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-0" prio=10 tid=0x00a39800 nid=0x1f in Object.wait() [0xddeff000..0xddeff970]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe5bcee08> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0xe5bcee08> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"RMI Scheduler(0)" daemon prio=10 tid=0x015ba000 nid=0x1e waiting on condition [0xddfff000..0xddfffaf0]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0xe5a0aa90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1927)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
        at java.lang.Thread.run(Thread.java:619)

"GC Daemon" daemon prio=2 tid=0x00cc9c00 nid=0x1c in Object.wait() [0xde1ff000..0xde1ffbf0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe59d6318> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0xe59d6318> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=10 tid=0x00775400 nid=0x1b in Object.wait() [0xde2ff000..0xde2ffb70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe59d6320> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xe59d6320> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x00a3a800 nid=0x1a runnable [0xde3ff000..0xde3ff8f0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xe59d6358> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x00130000 nid=0xb runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x0012e400 nid=0xa waiting on condition [0x00000000..0xdf7fed68]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0012d000 nid=0x9 waiting on condition [0x00000000..0xdf8fece8]
   java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" daemon prio=10 tid=0x00122000 nid=0x8 runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x0011ec00 nid=0x7 runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00105000 nid=0x6 runnable [0x00000000..0xdfbff7d8]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x000f3000 nid=0x5 in Object.wait() [0xdfcff000..0xdfcffa70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe55610c0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xe55610c0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000f2800 nid=0x4 in Object.wait() [0xdfdff000..0xdfdffaf0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xe55610e0> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0xe55610e0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x000f0000 nid=0x3 runnable

"VM Periodic Task Thread" prio=10 tid=0x00132000 nid=0xc waiting on condition

JNI global references: 11114