Here is a explanation of the issue my co-worker sent to the powers that be within my company. Please reply if you agree or disagree with his assessment.
---------------------------------
Overview:
Upon server startup there is a finite number of threads available to service user requests. When a thread comes in requesting a page that has not yet been viewed since the last time the server started, the server will block all other requestors to that page until the page has been compiled.
In order to compile the page the server forks a new process (not an in-process thread) that executes the compile. Upon completion of the compile (successful return code from the fork) the initiating thread is released to service the page request, as are all the additional blocked threads for that same page.
The error occurs when the forked process fails to return. From the system's perspective it is still running. It is important to differentiate this from a full failure in the compile process which would release the lock and display an error message. In this case the process does not return at all and the lock persists indefinitely.
Overtime additional requests for this orphaned page will each take a thread out of the available pool and place it in line behind the lock. Eventually all threads from the finite available pool are consumed waiting for the lock and the server becomes unresponsive and must be restarted.
Ugly Details follow:
Likely root cause: Known issue with JVM on Solaris 10 (exec hangs):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6671051
Likely solution: upgrade to at least java 1.6 update 7 (update 11 is latest version)
Possible work around: Precompiling JSP files will reduce but not eliminate calls to UnixProcess.forkAndExec.
Obviously we will have to do a great deal of testing in order to upgrade the JVM, but I think that an upgrade is the only actual solution. The calls to UnixProcess.forkAndExec are not isolated to JSP compilation, but are used throughout the application server and this bug can result in numerous other unexplained process hangs.
Analyzing the threads available on the application server we see the following:
##############################################################################
1 Thread executing the forked process:
Thread t_at_4193: (state = IN_NATIVE)
- java.lang.UNIXProcess.forkAndExec(byte[], byte[], int, byte[], int, byte[], boolean, java.io.FileDescriptor, java.io.FileDescriptor, java.io.FileDescriptor) @bci=1350864168 (Interpreted frame)
1 Thread in a WAITING state waiting on the return of the forked process:
Thread "http38183-Processor11" thread-id 4,133 thread-stateWAITINGWaiting on lock: java.lang.UNIXProcess_at_1aa11a8
498 Threads in a BLOCKED state waiting for the lock to be released by the WAITING thread:
Thread "http38183-Processor12" thread-id 4,134 thread-stateBLOCKEDWaiting on lock:
Owned by: http38183-Processor11 Id: 4,133
##############################################################################
This situation persisted over the course of several hours with no change.
Here is the location of the lock in Sun's source code (JSPServletWrapper.service()):
if (!options.getUsePrecompiled()
&& (options.getDevelopment() || firstTime)) {
// END S1AS 6181923
synchronized (this) {
firstTime = false;
// The following sets reload to true, if necessary
ctxt.compile();
}
} else {
if (compileException != null) {
// Throw cached compilation exception
throw compileException;
}
}
It is important to note that the lock itself is entirely appropriate. It would be incorrect to allow two simultaneous threads to compile the same jsp page twice. At issue is the fact that the forked process behind the lock does not return at all.
[Message sent by forum member 'jfaldmo' (jfaldmo)]
http://forums.java.net/jive/thread.jspa?messageID=324653