Thanks a lot Radim.
Looking at the stacktraces you provided, seems like the main problem
comes from the DB related code either GF resource pool or JDBC driver or
DB server itself.
For example hang1 and slow.server samples reports lots of worker threads
in WAIT state [1], hope folks responsible for ConnectionManager can
advice how to configure pool better to avoid this (may be increase pool
size and/or decrease timeout value)...
The slow.server2 report is similar, but has a stacktrace, which suggests
that problem could be related to the postgresql, one of the
possibilities is that postgresql is responding slowly and this slows
down entire GF server.
The forgot-what-is-wrong and hang3 are the same IMO, but I can't say
what's wrong there, all the worker threads are doing nothing and just
waiting for data from apache server...
WBR,
Alexey.
"TP-Processor32" daemon prio=5 tid=0x000000083d5b8800 nid=0x188ea in
Object.wait() [0x00007ffff9597000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000082aac1370> (a java.lang.Object)
at
com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:432)
- locked <0x000000082aac1370> (a java.lang.Object)
at
com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:242)
at
com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:167)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:236)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:192)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at
org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:232)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:337)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:817)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:746)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:939)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:679)
"TP-Processor31" daemon prio=5 tid=0x000000083e1e4000 nid=0x18ada
runnable [0x00007ffffb9ba000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:269)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at
org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at
org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:200)
at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:89)
at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:72)
at
com.sun.gjc.spi.DSManagedConnectionFactory.createManagedConnection(DSManagedConnectionFactory.java:113)
at
com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.createResource(LocalTxConnectorAllocator.java:87)
at
com.sun.enterprise.resource.pool.ConnectionPool.createSingleResource(ConnectionPool.java:907)
at
com.sun.enterprise.resource.pool.ConnectionPool.createResource(ConnectionPool.java:1185)
at
com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.addResource(RWLockDataStructure.java:98)
at
com.sun.enterprise.resource.pool.ConnectionPool.addResource(ConnectionPool.java:282)
at
com.sun.enterprise.resource.pool.ConnectionPool.createResourceAndAddToPool(ConnectionPool.java:1501)
at
com.sun.enterprise.resource.pool.ConnectionPool.createResources(ConnectionPool.java:944)
at
com.sun.enterprise.resource.pool.ConnectionPool.resizePoolAndGetNewResource(ConnectionPool.java:792)
at
com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:760)
at
com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632)
at
com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526)
at
com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
at
com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:242)
at
com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:167)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:236)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:186)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at
org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:232)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:337)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:817)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:746)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:939)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:679)
On 03/21/2012 11:36 AM, forums_at_java.net wrote:
> I am sending you promisted jstack dumps from hanged glassfish. There
> are 2
> problem types:
>
> 1. hanged server. Not responding to any http requests.
>
> 2. server runs but very slowly. This is server state after it has been
> overloaded for a while with Apache bench, after few minutes it
> recovers from
> hang but runs very slowly.
>
>
> --
>
> [Message sent by forum member 'rkolar02']
>
> View Post: http://forums.java.net/node/884437
>
>
>