users@glassfish.java.net

Re: Glassfish hangs after a few days

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 16 Sep 2009 11:55:06 -0400

Salut,

glassfish_at_javadesktop.org wrote:
> We are running on glassfish v2.1 and we have 1000 Acceptor Threads configured for https.

Is there any reason why you set up 1000 Acceptor Threads. Usually that
value should be based on the number of core/processor. Instead, you
should configure the thread-count value of request-processing element in
domain.xml to 1000.

After about 2 weeks, all of the https connections hang. We cannot
connect to the Glassfish application server via https when this happens.
We can connect via http, it works fine. Leading up to this problem we
see intermittent broken pipes and JDBC connectivity issues in the
server.log over a period of time. The theory is that the https
connections are not disconnecting when an network related error occurs.
  This eventually consumes all 1000 acceptor connector threads.
>
> After investigating the https connectivity issues we found Grizzly bug number 680 and we think it may be related to the https hang.
>
> Additionally, we may have a seperate unrelated application error consuming heap space. We found bug number 730 that may be related to the heap space memory error.
>

Hum the issue you are refering only apply to Grizzly 1.9 and up. The
Grizzly you are using use grizzly 1.0.x, which doesn't have those issues.

> Grizzly Bug 730
> -------------------
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=730
> ServletAdapter.destroy() doesn't destroy wrapped HttpServlet

TRhe grizzly Servlet Container is *not* used by GlassFish v2/v3.
GlassFish uses a modified version of Tomcat 5.

>
> Grizzly Bug 680
> -------------------
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=680
> SSL client may hang with 1.9.16 when send series packets to server then wait for a reply
> Fixed in Grizly 1.9.17 (August 12th 2009)
> Fixed Issue List: http://tinyurl.com/ohyjs4

>
> Grizly 1.9.18a
> ----------------
> https://grizzly.dev.java.net/
> https://grizzly.dev.java.net/issues/buglist.cgi?Submit+query=Submit+query&component=grizzly&issue_status=RESOLVED&issue_status=VERIFIED&issue_status=CLOSED&target_milestone=1.9.18&email1=&emailtype1=exact&emailassigned_to1=1&emailtype2=exact&emailreporter2=1&issueidtype=include&chfieldto=Now&short_desc_type=fulltext&long_desc_type=fulltext&issue_file_loc_type=fulltext
>
>
> We think upgrading to grizzly version 1.9.18a will solve this problem. We'd like to know if replacing the grizzly files in our current configuration would be supported.
> Netstat shows about 5,000 https connections in CLOSE_WAIT state.

Yes that's could be an issue, which JDK are you using? You should try
with the latest Grizzly 1.0.30 jar to see if the problem still occurs.
You can download it from here

    * http://is.gd/3llfs

Just add the jar to your classpath-prefix attribute in domain.xml

>
> TCP: IPv4
> Local Address Remote Address Swind Send-Q Rwind Recv-Q State
> -------------------- -------------------- ----- ------ ----- ------ -----------
> 172.18.244.177.443 172.18.244.5.9144 17520 0 33304 0 CLOSE_WAIT
> 172.18.244.177.443 172.18.244.6.41429 17520 0 33304 0 CLOSE_WAIT
> 172.18.244.177.443 172.18.244.5.51464 17520 0 33304 0 CLOSE_WAIT
> 172.18.244.177.443 172.18.244.6.20586 17520 0 33304 0 CLOSE_WAIT
> 172.18.244.177.443 172.18.244.6.55477 17520 0 33304 0 CLOSE_WAIT
>
> The logs show us the following types of errors:
>
> (1) javax.enterprise.system.stream.err in the httpSSLWorkerThread from (com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run)
> Caused by: java.net.SocketException: Broken pipe
>
> (2) javax.enterprise.system.stream.err in the httpSSLWorkerThread (grizzly.ssl.SSLWorkerThread)
> java.lang.NullPointerException

Do you have the complete stack trace?

>
> (3) javax.enterprise.system.stream.out in the httpSSLWorkerThread
> java.net.ConnectException: Connection refused
>
> (4) java.lang.OutOfMemoryError: Java heap space

That one won't help for sure.

>
> (5) java.net.SocketException: Invalid argument
> [Message sent by forum member 'britton_laroche' (britton_laroche_at_yahoo.com)]

This is a JDK issue:

   * http://bugs.sun.com/view_bug.do?bug_id=6693490

A+

-- Jeanfrancois


>
> http://forums.java.net/jive/thread.jspa?messageID=364125
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>