users@glassfish.java.net

Re: DAS and node instance disappear under moderate load

From: <glassfish_at_javadesktop.org>
Date: Tue, 26 May 2009 07:42:14 PDT

I think we found out... and Glassfish hasn't been guilty.

Indeed your suggestion to look at the OS made me dig in system logs. It turns out that the DAS and instance processes were killed by RedHat's [i]oom-killer[/i].

It's surprising because the monitoring and logs show there is swap room left, but it becomes an OS issue. The OOM-killer chooses the most memory-hungry processes for selective elimination, and, unsurprisingly, they turn out to be our server-side nodes and instances which are configured with a rather large heap.

We're going to try giving less heap to the DAS, though I am clueless as to how much memory it needs. That probably won't save our instances' JVMs: we still have to investigate, at the OS level, who's consuming memory, or tune how the oom-killer triggers...

Thanks for your answering; I leave this note in case someone ever searches for Glassfish DAs or instance process vanishing on RedHat (notice the dense usage of keywords together to get that indexed :o)
[Message sent by forum member 'jduprez' (jduprez)]

http://forums.java.net/jive/thread.jspa?messageID=347688