Internally GlassFish uses Tomcat for the servlet container. Tomcat has
pluggable "http servers" (I don't know the correct term, connectors?),
and it comes with one called Coyote. JBoss also uses Tomcat for the
servlet container, and Coyote as the "http server".
By default, GlassFish uses a different "http server" called Grizzly,
written by Sun. It is high performance and has a lot of features:
https://grizzly.dev.java.net/
Jeanfrancois at Sun is one of the main developers of Grizzly, and he
has been helping us on this thread. One of his ideas is to swap out
Grizzly for Coyote by opening domains/domain1/config/domain.xml,
searching for "<jvm-option>" (you will find a bunch, one per line), and
adding a new JVM option:
<jvm-option>-Dcom.sun.enterprise.web.useCoyoteConnector=true</jvm-option>
Restart GlassFish. If the problem still happens, then he will know it
is not Grizzly's fault. He also wants a thread dump while it is in a
state that does not work to see if there are deadlocked threads. I ran
this command from the command line:
asadmin generate-jvm-report --type=thread
Just now I realized that this command does not prompt for a username and
password on our production server. I should have been able to type:
asadmin generate-jvm-report --type=thread > dump.txt
Darn! I'll do that next time. As for other things we have tried... not
much. We got in touch with Microsoft to get our hands on some patches
they've been working on to solve the NP Pool issue. After we installed
it, we started going down every few days. Starting yesterday it has
been 2-3 times per day. I think it made the problem worse, but if
Jeanfrancois is right then that is just a coincidence because the
problem has nothing to do with the NP Pool leak in Windows.
Up until now, we (customer) and Sun have been under the impression that
the problem was Microsoft's fault (NP Pool leak) and that there is
nothing we can do until Microsoft fixes it or until we change OS's.
Your thread has enlightened everyone: it happens on Linux too, so only
one of three symptoms is caused by the leak. That's why there's a
flurry of activity in this thread now. We need to find out if the
problem is my application's fault, or GlassFish's.
Thanks,
Ryan
> Ryan, regarding your posts specifically I stand corrected. Could you please provide more detail as to what you are trying and exactly what JVM options you are changing? I am not at all familiar with what either Grizzly or Coyote are. I do know that I see Grizzly errors in my logs frequently.