We are running on glassfish v2.1 and we have 1000 Acceptor Threads configured for https. After about 2 weeks, all of the https connections hang. We cannot connect to the Glassfish application server via https when this happens. We can connect via http, it works fine. Leading up to this problem we see intermittent broken pipes and JDBC connectivity issues in the server.log over a period of time. The theory is that the https connections are not disconnecting when an network related error occurs. This eventually consumes all 1000 acceptor connector threads.
After investigating the https connectivity issues we found Grizzly bug number 680 and we think it may be related to the https hang.
Additionally, we may have a seperate unrelated application error consuming heap space. We found bug number 730 that may be related to the heap space memory error.
Grizzly Bug 730
-------------------
https://grizzly.dev.java.net/issues/show_bug.cgi?id=730
ServletAdapter.destroy() doesn't destroy wrapped HttpServlet
Grizzly Bug 680
-------------------
https://grizzly.dev.java.net/issues/show_bug.cgi?id=680
SSL client may hang with 1.9.16 when send series packets to server then wait for a reply
Fixed in Grizly 1.9.17 (August 12th 2009)
Fixed Issue List:
http://tinyurl.com/ohyjs4
Grizly 1.9.18a
----------------
https://grizzly.dev.java.net/
https://grizzly.dev.java.net/issues/buglist.cgi?Submit+query=Submit+query&component=grizzly&issue_status=RESOLVED&issue_status=VERIFIED&issue_status=CLOSED&target_milestone=1.9.18&email1=&emailtype1=exact&emailassigned_to1=1&emailtype2=exact&emailreporter2=1&issueidtype=include&chfieldto=Now&short_desc_type=fulltext&long_desc_type=fulltext&issue_file_loc_type=fulltext
We think upgrading to grizzly version 1.9.18a will solve this problem. We'd like to know if replacing the grizzly files in our current configuration would be supported.
Netstat shows about 5,000 https connections in CLOSE_WAIT state.
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -----------
172.18.244.177.443 172.18.244.5.9144 17520 0 33304 0 CLOSE_WAIT
172.18.244.177.443 172.18.244.6.41429 17520 0 33304 0 CLOSE_WAIT
172.18.244.177.443 172.18.244.5.51464 17520 0 33304 0 CLOSE_WAIT
172.18.244.177.443 172.18.244.6.20586 17520 0 33304 0 CLOSE_WAIT
172.18.244.177.443 172.18.244.6.55477 17520 0 33304 0 CLOSE_WAIT
The logs show us the following types of errors:
(1) javax.enterprise.system.stream.err in the httpSSLWorkerThread from (com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run)
Caused by: java.net.SocketException: Broken pipe
(2) javax.enterprise.system.stream.err in the httpSSLWorkerThread (grizzly.ssl.SSLWorkerThread)
java.lang.NullPointerException
(3) javax.enterprise.system.stream.out in the httpSSLWorkerThread
java.net.ConnectException: Connection refused
(4) java.lang.OutOfMemoryError: Java heap space
(5) java.net.SocketException: Invalid argument
[Message sent by forum member 'britton_laroche' (britton_laroche_at_yahoo.com)]
http://forums.java.net/jive/thread.jspa?messageID=364125