If you have a Java heap of 3.5Gb you have left virtually no space in the
process virtual address space for anything else. The OutOfMemory error
is most likely caused because there is no space left to allow the
creation of any additional threads - having 9Gb of swap doesn't help you
in this case. The error says:
stack: java.lang.OutOfMemoryError
i.e. insufficient memory to create the stack for a new thread, this is
in the processes memory address space.
Note that you are limited to using 32-bit JVMs with AS 7.x, something
that isn't the case with glassfish/Sun AS 9.x.
You should probably look to reducing the size of your virtual heap
significantly to allow the creation of the additional threads. If your
application needs 3.5Gb or more of Java object heap to function then you
would have to look to moving to a clustered setup and spread your load
over multiple instances - something possible with AS 7.1.
If you are using Oracle's OCI JDBC driver (i.e.
jdbc:oracle:oci:@some-alias in the pool definitions) there were issues
with it that could lead to leaks of native memory which could help
contribute to the problem). Using a type-4 driver
(jdbc:oracle:thin:@hostname:port:sid) would remove this from your
trouble shooting.
The JDBC pool issues could be for a number of reasons, and should be
unconnected to the memory issue ... you should really ensure that you
are on the latest update version, its not clear if you are on AS 7.0 or
7.1, currently the latest updates are 7.0 UR10 and 7.1 UR6, as these
contained fixes that could lead to leaks of connections in the pool.
You should also enable connection validation and change it to use table
based validation on the table DUAL instead of auto-commit which is
worthless because of the way the Oracle JDBC driver optimises its
communication with the back end database.
If none of the above suggestions help then I'd suggest you contact Sun
Support and raise a case with them.
Finally AS 7.x is in Sun's EOL process and there will be no further
sustaining or bug fix support from the end of October. You should be
looking at migrating to a fully supported version, e.g. AS 8.x or AS 9.x.
Regards,
Steve
glassfish_at_javadesktop.org wrote:
> The production site is running on solaris machine so its not possible to use profiler there.
>
> The logs for sun server depicts some error as follows -
>
> [22/Aug/2008:04:16:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
> [22/Aug/2008:04:16:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
> [22/Aug/2008:04:17:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
> [22/Aug/2008:04:17:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
> [22/Aug/2008:04:18:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
> [22/Aug/2008:04:18:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:19:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:19:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:20:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:20:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:21:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:21:26] WARNING (22501): for host 172.16.0.108 trying to GET /robots.txt, send-file reports: HTTP4142: can't find /var/opt/SUNWappserver7/domains/domain1/server1/applications/j2ee-modules/AAI_CMSPROD_1/robots.txt (File not found)
> [22/Aug/2008:04:21:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:22:15] WARNING (22501): CORE3283: stderr: java.lang.OutOfMemoryError
> [22/Aug/2008:04:22:20] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
> [22/Aug/2008:04:22:28] SEVERE (22501): CORE4007: Internal error: Unexpected Java exception thrown (java.lang.OutOfMemoryError, no description), stack: java.lang.OutOfMemoryError
>
> The above logs depict that as soon as PoolSize becomes 31 from 32 , FreeResources also turn 0 from 1 and this throws OutOfMemoryError. Now the server needs to be restarted which makes the PoolSize again as 8 and the site starts working again.
>
> Does this have to do anything with the entries in server.xml which is as follows in my case -
>
> <jdbc-connection-pool steady-pool-size="8" max-pool-size="32" max-wait-time-in-millis="60000" pool-resize-quantity="2" idle-timeout-in-seconds="300" is-isolation-level-guaranteed="false" is-connection-validation-required="false" connection-validation-method="auto-commit" fail-all-connections="false" datasource-classname="oracle.jdbc.pool.OracleDataSource" res-type="javax.sql.XADataSource">
> [Message sent by forum member 'puneetchachra' (puneetchachra)]
>
> http://forums.java.net/jive/thread.jspa?messageID=294811
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>