My brilliant co-worker developer was able to read the threaddumps/jstack files. He also download the glassfish code to see the line that was blocking the threads. He was able to determine that the block was happening when the JVM was trying to compile a jsp file. I killed the forked process (saw another process start and then it went away) and the server appears to be responding again. A new jsp file was in the generated directory structure.
There are still 138 blocked threads. Here are the two messages from the thread dump. I have also atteched the files if you are interested.
Thread "RMI ConnectionExpiration-[192.168.58.45:50336,com.sun.appserv.management.client.AdminRMISSLClientSocketFactory_at_14471]" thread-id 28,307 thread-stateTIMED_WAITING
at: java.lang.Thread.sleep(Native Method)
at: sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)
at: java.lang.Thread.run(Thread.java:595)
Thread "http38183-Processor32" thread-id 26,502 thread-stateWAITINGWaiting on lock: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable_at_c37726
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:474)
at: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:639)
at: java.lang.Thread.run(Thread.java:595)
[Message sent by forum member 'jfaldmo' (jfaldmo)]
http://forums.java.net/jive/thread.jspa?messageID=324643