users@glassfish.java.net

Re: Glassfish scalability issue

From: <glassfish_at_javadesktop.org>
Date: Thu, 19 Jul 2007 04:21:29 PDT

Hi,

Thanks for the very quick reply.

> 1) Do you have any firewall configured ?

I have no firewall setup between the machines running server, db or test tools.

> 2) Could you post the pool configuration ?

Connection Pool Settings:
<jdbc-connection-pool allow-non-component-callers="false" associate-with-thread="false" connection-creation-retry-attempts="20" connection-creation-retry-interval-in-seconds="3" connection-leak-reclaim="true" connection-leak-timeout-in-seconds="30" connection-validation-method="table" datasource-classname="oracle.jdbc.pool.OracleDataSource" fail-all-connections="true" idle-timeout-in-seconds="300" is-connection-validation-required="true" is-isolation-level-guaranteed="false" lazy-connection-association="false" lazy-connection-enlistment="false" match-connections="false" max-connection-usage-count="0" max-pool-size="32" max-wait-time-in-millis="60000" name="oraRpt2Local" non-transactional-connections="false" pool-resize-quantity="2" res-type="javax.sql.DataSource" statement-timeout-in-seconds="-1" steady-pool-size="8" validate-atmost-once-period-in-seconds="0" validation-table-name="dual" wrap-jdbc-objects="true">
      <property name="DataSourceName" value="OracleDataSource"/>
      <property name="ImplicitCachingEnabled" value="false"/>
      <property name="NetworkProtocol" value="tcp"/>
      <property name="Password" value="rpt2"/>
      <property name="LoginTimeout" value="0"/>
      <property name="User" value="rpt2"/>
      <property name="ExplicitCachingEnabled" value="false"/>
      <property name="PortNumber" value="0"/>
      <property name="MaxStatements" value="0"/>
      <property name="URL" value="jdbc:oracle:thin:@tstappsvr1:11521:s10"/>
    </jdbc-connection-pool>

When running my last test I changed the fail-all-connections="false" setting to "true". When doing this the INFO message in my previous post did not appear anymore. But before the same SEVERE call stack from previous this WARNING happened:

#|2007-07-19T11:51:42.390-0700|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=16;_ThreadName=httpSSLWorkerThread-8080-1;_RequestID=403ad4ac-e7c7-4ba2-8830-d67c8f9e7db5;|java.sql.SQLException: Io exception: Connection reset
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1039)
        at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:291)
        at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:213)
        at oracle.jdbc.driver.ScrollableResultSet.cacheRowAt(ScrollableResultSet.java:1743)
        at oracle.jdbc.driver.ScrollableResultSet.isValidRow(ScrollableResultSet.java:1726)
        at oracle.jdbc.driver.ScrollableResultSet.next(ScrollableResultSet.java:315)
        at com.sun.gjc.spi.base.ResultSetWrapper.next(ResultSetWrapper.java:82)
        at com.retaininternational.ejb.GetRtnLookupDataBean.multiIncLookup(GetRtnLookupDataBean.java:212)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
        at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
        at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2884)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3975)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy26.multiIncLookup(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__GetRtnLookupDataRemote_Remote_DynamicStub.multiIncLookup(com/retaininternational/remoteejb/__GetRtnLookupDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._GetRtnLookupDataRemote_Wrapper.multiIncLookup(com/retaininternational/remoteejb/_GetRtnLookupDataRemote_Wrapper.java)
        at com.retain.ejbclient.RtnEJBLookupData.multiIncLookup(RtnEJBLookupData.java:42)
        at com.retain.ws.RtnBkgWSClient.multiIncLookup(RtnBkgWSClient.java:154)
        at com.retain.ajax.servlets.MultiIncLookup.processRequest(MultiIncLookup.java:53)
        at com.retain.ajax.servlets.MultiIncLookup.doPost(MultiIncLookup.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)


> 3) When the appserver is hung, could you get a thread
> dump (jstack) of appserver process.

jstack output: (Sorry about the large content, could not find a way of attaching the output file)


2007-07-19 11:52:27
Full thread dump Java HotSpot(TM) Client VM (1.6.0_02-b05 mixed mode):

"httpSSLWorkerThread-8080-4" daemon prio=10 tid=0x2f947000 nid=0x480 waiting for monitor entry [0x300fd000..0x300ffb18]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at com.sun.enterprise.resource.AbstractResourcePool.transactionCompleted(AbstractResourcePool.java:1111)
        - waiting to lock <0x0f7b1d88> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.PoolManagerImpl.transactionCompleted(PoolManagerImpl.java:393)
        at com.sun.enterprise.resource.PoolManagerImpl$SynchronizationListener.afterCompletion(PoolManagerImpl.java:512)
        at com.sun.enterprise.distributedtx.J2EETransaction.commit(J2EETransaction.java:491)
        at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.commit(J2EETransactionManagerOpt.java:371)
        at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:3781)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:3560)
        at com.sun.ejb.containers.StatefulSessionContainer.postInvokeTx(StatefulSessionContainer.java:2714)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1343)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1305)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:210)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy30.getWallchartData(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__WallchartDataRemote_Remote_DynamicStub.getWallchartData(com/retaininternational/remoteejb/__WallchartDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._WallchartDataRemote_Wrapper.getWallchartData(com/retaininternational/remoteejb/_WallchartDataRemote_Wrapper.java)
        at com.retain.ws.RtnBkgWSClient.getWallchartData(RtnBkgWSClient.java:109)
        at com.retain.component.wallchart.RtnWallchartBean.getWallchartData(RtnWallchartBean.java:74)
        at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
        at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)
        at com.sun.faces.application.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:77)
        at org.apache.shale.remoting.impl.MethodBindingProcessor.process(MethodBindingProcessor.java:122)
        at org.apache.shale.remoting.faces.RemotingPhaseListener.afterPhase(RemotingPhaseListener.java:114)
        at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:280)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
        at com.sun.faces.extensions.avatar.lifecycle.PartialTraversalLifecycle.execute(PartialTraversalLifecycle.java:80)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:244)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
        at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:266)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8080-3" daemon prio=10 tid=0x2fb81c00 nid=0x438 runnable [0x2fe3e000..0x2fe3fb98]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at oracle.net.ns.Packet.receive(Unknown Source)
        at oracle.net.ns.DataPacket.receive(Unknown Source)
        at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1029)
        at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:291)
        - locked <0x0f7b93a8> (a oracle.jdbc.driver.T4CConnection)
        at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:213)
        - locked <0x12720e80> (a oracle.jdbc.driver.OracleResultSetImpl)
        at oracle.jdbc.driver.ScrollableResultSet.cacheRowAt(ScrollableResultSet.java:1743)
        at oracle.jdbc.driver.ScrollableResultSet.isValidRow(ScrollableResultSet.java:1726)
        at oracle.jdbc.driver.ScrollableResultSet.next(ScrollableResultSet.java:315)
        - locked <0x12720ea0> (a oracle.jdbc.driver.ScrollableResultSet)
        at com.sun.gjc.spi.base.ResultSetWrapper.next(ResultSetWrapper.java:82)
        at com.retaininternational.ejb.GetRtnLookupDataBean.multiIncLookup(GetRtnLookupDataBean.java:212)
        at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
        at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
        at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2884)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3975)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy26.multiIncLookup(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__GetRtnLookupDataRemote_Remote_DynamicStub.multiIncLookup(com/retaininternational/remoteejb/__GetRtnLookupDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._GetRtnLookupDataRemote_Wrapper.multiIncLookup(com/retaininternational/remoteejb/_GetRtnLookupDataRemote_Wrapper.java)
        at com.retain.ejbclient.RtnEJBLookupData.multiIncLookup(RtnEJBLookupData.java:42)
        at com.retain.ws.RtnBkgWSClient.multiIncLookup(RtnBkgWSClient.java:154)
        at com.retain.ajax.servlets.MultiIncLookup.processRequest(MultiIncLookup.java:53)
        at com.retain.ajax.servlets.MultiIncLookup.doPost(MultiIncLookup.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8080-2" daemon prio=10 tid=0x2fb5e800 nid=0x538 runnable [0x2fdee000..0x2fdefc18]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at oracle.net.ns.Packet.receive(Unknown Source)
        at oracle.net.ns.DataPacket.receive(Unknown Source)
        at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1029)
        at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:291)
        - locked <0x0f9b5258> (a oracle.jdbc.driver.T4CConnection)
        at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:213)
        - locked <0x13d46110> (a oracle.jdbc.driver.OracleResultSetImpl)
        at oracle.jdbc.driver.ScrollableResultSet.cacheRowAt(ScrollableResultSet.java:1743)
        at oracle.jdbc.driver.ScrollableResultSet.isValidRow(ScrollableResultSet.java:1726)
        at oracle.jdbc.driver.ScrollableResultSet.next(ScrollableResultSet.java:315)
        - locked <0x13c8dbc0> (a oracle.jdbc.driver.ScrollableResultSet)
        at com.sun.gjc.spi.base.ResultSetWrapper.next(ResultSetWrapper.java:82)
        at com.retaininternational.ejb.GetRtnLookupDataBean.multiIncLookup(GetRtnLookupDataBean.java:212)
        at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
        at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
        at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2884)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3975)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy26.multiIncLookup(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__GetRtnLookupDataRemote_Remote_DynamicStub.multiIncLookup(com/retaininternational/remoteejb/__GetRtnLookupDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._GetRtnLookupDataRemote_Wrapper.multiIncLookup(com/retaininternational/remoteejb/_GetRtnLookupDataRemote_Wrapper.java)
        at com.retain.ejbclient.RtnEJBLookupData.multiIncLookup(RtnEJBLookupData.java:42)
        at com.retain.ws.RtnBkgWSClient.multiIncLookup(RtnBkgWSClient.java:154)
        at com.retain.ajax.servlets.MultiIncLookup.processRequest(MultiIncLookup.java:53)
        at com.retain.ajax.servlets.MultiIncLookup.doPost(MultiIncLookup.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"Timer-6" daemon prio=10 tid=0x30dd6000 nid=0xe4c waiting for monitor entry [0x31daf000..0x31dafc98]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:992)
        - waiting to lock <0x0f7b1d88> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344)
        - locked <0x0f7b2078> (a java.lang.Object)
        at com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62)
        at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1471)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"Timer-5" prio=6 tid=0x2f9ab000 nid=0xdcc in Object.wait() [0x30d5f000..0x30d5fd18]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x0e826158> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"DestroyJavaVM" prio=6 tid=0x0029c400 nid=0xcb0 waiting on condition [0x00000000..0x0088fd50]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Timer-4" daemon prio=6 tid=0x0092a400 nid=0xdc8 in Object.wait() [0x30d0f000..0x30d0fd98]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x0e826288> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=6 tid=0x00937800 nid=0xdbc waiting on condition [0x30cbf000..0x30cbfb18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/visualwallchart]]" daemon prio=6 tid=0x2f9bbc00 nid=0xdb8 waiting on condition [0x30c6f000..0x30c6fb98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/amserver]]" daemon prio=6 tid=0x2f9dac00 nid=0xdb4 waiting on condition [0x30c1f000..0x30c1fc18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]" daemon prio=6 tid=0x2f9d6c00 nid=0xdb0 waiting on condition [0x30bcf000..0x30bcfc98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[]]" daemon prio=6 tid=0x2f9fb400 nid=0xdac waiting on condition [0x30b7f000..0x30b7fd18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]" daemon prio=6 tid=0x2f94b800 nid=0xda8 waiting on condition [0x30b2f000..0x30b2fd98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[/web1]]" daemon prio=6 tid=0x2f9eb000 nid=0xda4 waiting on condition [0x30adf000..0x30adfa18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon prio=6 tid=0x2f9d8000 nid=0xda0 waiting on condition [0x30a8f000..0x30a8fa98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"httpWorkerThread-4848-1" daemon prio=10 tid=0x2f982800 nid=0xd90 in Object.wait() [0x3099f000..0x3099fc18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0e1fe4e0> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
        - locked <0x0e1fe4e0> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)

   Locked ownable synchronizers:
        - None

"httpWorkerThread-4848-0" daemon prio=10 tid=0x2f982000 nid=0xd8c in Object.wait() [0x3094f000..0x3094fc98]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0e1fe4e0> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
        - locked <0x0e1fe4e0> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)

   Locked ownable synchronizers:
        - None

"SelectorThread-4848" prio=6 tid=0x2f980c00 nid=0xd88 runnable [0x308ff000..0x308ffd18]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x0e1ff4b0> (a sun.nio.ch.Util$1)
        - locked <0x0e1ff4a0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0e1ff280> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:133)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1276)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1223)
        - locked <0x0e1fd440> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1186)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1162)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8181-1" daemon prio=10 tid=0x2f980000 nid=0xd84 in Object.wait() [0x308af000..0x308afa18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0e133188> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
        - locked <0x0e133188> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8181-0" daemon prio=10 tid=0x2f97f800 nid=0xd80 in Object.wait() [0x3085f000..0x3085fd98]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0e133188> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
        - locked <0x0e133188> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)

   Locked ownable synchronizers:
        - None

"SelectorThread-8181" prio=6 tid=0x2f995800 nid=0xd7c runnable [0x3080f000..0x3080fa98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x0e134168> (a sun.nio.ch.Util$1)
        - locked <0x0e134158> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0e133f38> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:133)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1276)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1223)
        - locked <0x0e129530> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1186)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1162)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8080-1" daemon prio=10 tid=0x2f98d800 nid=0xd78 runnable [0x3005e000..0x3005fb18]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at oracle.net.ns.Packet.receive(Unknown Source)
        at oracle.net.ns.DataPacket.receive(Unknown Source)
        at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1029)
        at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:291)
        - locked <0x0f9b4e68> (a oracle.jdbc.driver.T4CConnection)
        at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:213)
        - locked <0x071804c0> (a oracle.jdbc.driver.OracleResultSetImpl)
        at oracle.jdbc.driver.ScrollableResultSet.cacheRowAt(ScrollableResultSet.java:1743)
        at oracle.jdbc.driver.ScrollableResultSet.isValidRow(ScrollableResultSet.java:1726)
        at oracle.jdbc.driver.ScrollableResultSet.next(ScrollableResultSet.java:315)
        - locked <0x071804e0> (a oracle.jdbc.driver.ScrollableResultSet)
        at com.sun.gjc.spi.base.ResultSetWrapper.next(ResultSetWrapper.java:82)
        at com.retaininternational.ejb.GetRtnLookupDataBean.multiIncLookup(GetRtnLookupDataBean.java:212)
        at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
        at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
        at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2884)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3975)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy26.multiIncLookup(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__GetRtnLookupDataRemote_Remote_DynamicStub.multiIncLookup(com/retaininternational/remoteejb/__GetRtnLookupDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._GetRtnLookupDataRemote_Wrapper.multiIncLookup(com/retaininternational/remoteejb/_GetRtnLookupDataRemote_Wrapper.java)
        at com.retain.ejbclient.RtnEJBLookupData.multiIncLookup(RtnEJBLookupData.java:42)
        at com.retain.ws.RtnBkgWSClient.multiIncLookup(RtnBkgWSClient.java:154)
        at com.retain.ajax.servlets.MultiIncLookup.processRequest(MultiIncLookup.java:53)
        at com.retain.ajax.servlets.MultiIncLookup.doPost(MultiIncLookup.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8080-0" daemon prio=10 tid=0x2f972800 nid=0xd74 waiting for monitor entry [0x3000e000..0x3000fb98]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:1179)
        - waiting to lock <0x0f7b93a8> (a oracle.jdbc.driver.T4CConnection)
        at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:284)
        at com.sun.enterprise.resource.LocalTxConnectorAllocator.destroyResource(LocalTxConnectorAllocator.java:114)
        at com.sun.enterprise.resource.AbstractResourcePool.destroyResource(AbstractResourcePool.java:907)
        at com.sun.enterprise.resource.AbstractResourcePool.emptyPool(AbstractResourcePool.java:1433)
        - locked <0x0f7b1d88> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.AbstractResourcePool.doFailAllConnectionsProcessing(AbstractResourcePool.java:1067)
        at com.sun.enterprise.resource.AbstractResourcePool.resourceErrorOccurred(AbstractResourcePool.java:1016)
        - locked <0x0f7b1d88> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.PoolManagerImpl.putbackResourceToPool(PoolManagerImpl.java:432)
        at com.sun.enterprise.resource.PoolManagerImpl.resourceErrorOccurred(PoolManagerImpl.java:405)
        at com.sun.enterprise.resource.LocalTxConnectionEventListener.connectionErrorOccurred(LocalTxConnectionEventListener.java:81)
        at com.sun.gjc.spi.ManagedConnection.connectionErrorOccurred(ManagedConnection.java:673)
        at com.sun.gjc.spi.ManagedConnectionFactory.getInvalidConnections(ManagedConnectionFactory.java:259)
        at com.sun.enterprise.resource.AbstractConnectorAllocator.getInvalidConnections(AbstractConnectorAllocator.java:99)
        at com.sun.enterprise.resource.AbstractConnectorAllocator.isConnectionValid(AbstractConnectorAllocator.java:111)
        at com.sun.enterprise.resource.AbstractResourcePool.isConnectionValid(AbstractResourcePool.java:685)
        at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:637)
        at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594)
        at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443)
        at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
        at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
        at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
        at com.retaininternational.struct.RtnStructUtils.getDbConnectiont(RtnStructUtils.java:181)
        - locked <0x257534e8> (a java.lang.Class for com.retaininternational.struct.RtnStructUtils)
        at com.retaininternational.sql.RtnSelectSql.getResultSetFromSql(RtnSelectSql.java:153)
        at com.retaininternational.ejb.GetRtnLookupDataBean.multiIncLookup(GetRtnLookupDataBean.java:209)
        at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
        at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
        at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2884)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3975)
        at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203)
        at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
        at $Proxy26.multiIncLookup(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
        at com.retaininternational.remoteejb.__GetRtnLookupDataRemote_Remote_DynamicStub.multiIncLookup(com/retaininternational/remoteejb/__GetRtnLookupDataRemote_Remote_DynamicStub.java)
        at com.retaininternational.remoteejb._GetRtnLookupDataRemote_Wrapper.multiIncLookup(com/retaininternational/remoteejb/_GetRtnLookupDataRemote_Wrapper.java)
        at com.retain.ejbclient.RtnEJBLookupData.multiIncLookup(RtnEJBLookupData.java:42)
        at com.retain.ws.RtnBkgWSClient.multiIncLookup(RtnBkgWSClient.java:154)
        at com.retain.ajax.servlets.MultiIncLookup.processRequest(MultiIncLookup.java:53)
        at com.retain.ajax.servlets.MultiIncLookup.doPost(MultiIncLookup.java:73)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"SelectorThread-8080" prio=6 tid=0x2f947c00 nid=0xd70 runnable [0x2ffbf000..0x2ffbfc18]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x0e2fb5d0> (a sun.nio.ch.Util$1)
        - locked <0x0e2fb5c0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0e2e7350> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:133)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1276)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1223)
        - locked <0x0e2dbf90> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1186)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1162)

   Locked ownable synchronizers:
        - None

"Thread-18" prio=6 tid=0x2f17c000 nid=0xd68 in Object.wait() [0x307bf000..0x307bfd18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0dff44d8> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:817)
        - locked <0x0dff44d8> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)

   Locked ownable synchronizers:
        - None

"p: thread-pool-1; w: 2" daemon prio=6 tid=0x2f146c00 nid=0xd64 runnable [0x3076f000..0x3076fd98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        - locked <0x0d724220> (a java.lang.Object)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        - locked <0x0d724200> (a java.lang.Object)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:188)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:168)
        - locked <0x0d716250> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
        at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:174)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:221)
        at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:72)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:524)

   Locked ownable synchronizers:
        - None

"p: thread-pool-1; w: 1" daemon prio=6 tid=0x2f146400 nid=0xd60 runnable [0x3071f000..0x3071fa18]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        - locked <0x0d7244c8> (a java.lang.Object)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        - locked <0x0d7244a8> (a java.lang.Object)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:188)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:168)
        - locked <0x0d716490> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
        at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:174)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
        at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:221)
        at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:72)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:524)

   Locked ownable synchronizers:
        - None

"SelectorThread" daemon prio=6 tid=0x2f162000 nid=0xd5c runnable [0x306cf000..0x306cfa98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x0df23738> (a sun.nio.ch.Util$1)
        - locked <0x0df23748> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0df236c0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:133)
        at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:257)

   Locked ownable synchronizers:
        - None

"RMI RenewClean-[10.20.30.80:1217]" daemon prio=6 tid=0x2f7ec000 nid=0xd48 in Object.wait() [0x3062f000..0x3062fd18]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0dd629e8> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-0" daemon prio=6 tid=0x2f7f8000 nid=0xd40 runnable [0x3058f000..0x3058fa18]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        - locked <0x0dd62b60> (a java.lang.Object)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        - locked <0x0dd62b68> (a java.lang.Object)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:188)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:168)
        - locked <0x0dd62b90> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
        at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:174)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-8686" daemon prio=6 tid=0x2f7f7800 nid=0xd3c runnable [0x3053f000..0x3053fa98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        - locked <0x0d725228> (a java.lang.Object)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        - locked <0x0d725208> (a java.lang.Object)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:188)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:168)
        - locked <0x0d718830> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
        at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:174)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"Thread-13" prio=6 tid=0x2f7b7800 nid=0xd30 waiting on condition [0x3044f000..0x3044fc18]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0dc48468> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
        at com.sun.enterprise.management.support.LoaderRegThread.process(LoaderRegThread.java:243)
        at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:154)

   Locked ownable synchronizers:
        - None

"Thread-12" prio=6 tid=0x2f7b7000 nid=0xd2c waiting on condition [0x303ff000..0x303ffc98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.enterprise.management.support.LoaderBase.mySleep(LoaderBase.java:241)
        at com.sun.enterprise.management.support.Loader$DeferredRegistrationThread.run(Loader.java:389)

   Locked ownable synchronizers:
        - None

"Timer-3" daemon prio=6 tid=0x2f795000 nid=0xd28 in Object.wait() [0x303af000..0x303afd18]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0dc0e198> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x0dc0e198> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"Timer-2" prio=6 tid=0x2f747c00 nid=0xd24 in Object.wait() [0x3035f000..0x3035fd98]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0dc0e220> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x0dc0e220> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"Timer-1" prio=6 tid=0x2f796400 nid=0xd20 in Object.wait() [0x3030f000..0x3030fa18]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x0dc0e2a8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"Thread-8" prio=6 tid=0x2f7afc00 nid=0xd1c waiting on condition [0x302bf000..0x302bfa98]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:151)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"RMI RenewClean-[10.20.30.80:1197,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_a]" daemon prio=6 tid=0x2f780000 nid=0xd18 in Object.wait() [0x3026f000..0x3026fb18]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0dbbfce0> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"Timer-0" prio=6 tid=0x2f6a1400 nid=0xd10 in Object.wait() [0x301cf000..0x301cfc18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0db22e38> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x0db22e38> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None

"RMI Scheduler(0)" daemon prio=6 tid=0x2f692c00 nid=0xd04 waiting on condition [0x3014f000..0x3014fc98]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0d7203c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"Thread-5" prio=6 tid=0x2f645000 nid=0xcf4 runnable [0x300af000..0x300afd98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:274)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:256)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:137)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x0d6fa988> (a sun.nio.ch.Util$1)
        - locked <0x0d6fa998> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0d6fa910> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
        at com.sun.enterprise.server.ss.provider.ASSelector.select(ASSelector.java:141)
        at com.sun.enterprise.server.ss.ASSocketService$EntryPointThread.run(ASSocketService.java:523)

   Locked ownable synchronizers:
        - None

"GC Daemon" daemon prio=2 tid=0x2f633400 nid=0xce4 in Object.wait() [0x2ff6f000..0x2ff6fb98]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0d703d70> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0x0d703d70> (a sun.misc.GC$LatencyLock)

   Locked ownable synchronizers:
        - None

"RMI Reaper" prio=6 tid=0x2f633000 nid=0xce0 in Object.wait() [0x2ff1f000..0x2ff1fc18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0d703de8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0d703de8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-0" daemon prio=6 tid=0x2f626400 nid=0xcdc runnable [0x2fecf000..0x2fecfc98]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        - locked <0x0d703ef0> (a java.lang.Object)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        - locked <0x0d703ef8> (a java.lang.Object)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.acceptSocket(ASServerSocket.java:188)
        at com.sun.enterprise.server.ss.provider.ASServerSocket.accept(ASServerSocket.java:168)
        - locked <0x0d703f20> (a com.sun.enterprise.server.ss.provider.ASServerSocket)
        at com.sun.enterprise.server.ss.provider.ASServerSocketImpl.accept(ASServerSocketImpl.java:174)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
        - None

"Low Memory Detector" daemon prio=6 tid=0x00925000 nid=0xccc runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"CompilerThread0" daemon prio=10 tid=0x00920800 nid=0xcc8 waiting on condition [0x00000000..0x2ee7f79c]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Attach Listener" daemon prio=10 tid=0x0091f400 nid=0xcc4 waiting on condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" daemon prio=10 tid=0x0091e400 nid=0xcc0 runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" daemon prio=8 tid=0x0090f000 nid=0xcbc in Object.wait() [0x2ed8f000..0x2ed8fc98]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0d4c4d08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

   Locked ownable synchronizers:
        - None

"Reference Handler" daemon prio=10 tid=0x0090a800 nid=0xcb8 in Object.wait() [0x2ed3f000..0x2ed3fd18]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x0d4c4d98> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
        - None

"VM Thread" prio=10 tid=0x00907800 nid=0xcb4 runnable

"VM Periodic Task Thread" prio=10 tid=0x00926800 nid=0xcd0 waiting on condition

JNI global references: 1301
[Message sent by forum member 'hansflhotmailcom' (hansflhotmailcom)]

http://forums.java.net/jive/thread.jspa?messageID=227261