The production site is running on solaris machine so its not possible to use profiler there.
The logs for sun server depicts some error as follows -
[22/Aug/2008:04:16:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:16:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:17:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:17:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:18:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:18:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:19:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:19:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:20:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:20:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:21:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:21:26] WARNING (22501): for host 172.16.0.108 trying to GET /robots.txt, send-file reports: HTTP4142: can't find /var/opt/SUNWappserver7/domains/domain1/server1/applications/j2ee-modules/AAI_CMSPROD_1/robots.txt (File not found)
[22/Aug/2008:04:21:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:22:15] WARNING (22501): CORE3283: stderr: java.lang.OutOfMemoryError
[22/Aug/2008:04:22:20] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:22:28] SEVERE (22501): CORE4007: Internal error: Unexpected Java exception thrown (java.lang.OutOfMemoryError, no description), stack: java.lang.OutOfMemoryError
The above logs depict that as soon as PoolSize becomes 31 from 32 , FreeResources also turn 0 from 1 and this throws OutOfMemoryError. Now the server needs to be restarted which makes the PoolSize again as 8 and the site starts working again.
Does this have to do anything with the entries in server.xml which is as follows in my case -
<jdbc-connection-pool steady-pool-size="8" max-pool-size="32" max-wait-time-in-millis="60000" pool-resize-quantity="2" idle-timeout-in-seconds="300" is-isolation-level-guaranteed="false" is-connection-validation-required="false" connection-validation-method="auto-commit" fail-all-connections="false" datasource-classname="oracle.jdbc.pool.OracleDataSource" res-type="javax.sql.XADataSource">
[Message sent by forum member 'puneetchachra' (puneetchachra)]
http://forums.java.net/jive/thread.jspa?messageID=294811