We've recently migrated our business app from WLS to GF v2.1 and just experienced the following:
- GF started to slow down at a point where most of the new incoming requests hang.
- netstat showed about 350 tcp connections with CLOSE_WAIT status.
- admin console was not accessible (hang).
- in attempt to shutdown the domain, asadmin stop-domain returned with success but it didn't actually terminate the process listening the http port of the given domain (had to manually kill the process).
- Web application deployed on the given domain have relatively light traffic over all (150-200 users a day, 5-10 concurrent users).
- happened on the 6th day after we went live.
- no errors/exceptions/nothing abnormal in the server log.
We did:
- look at "Thread: https (8181) not responding - v2_rc1_b58" and reviewed the recommendation of configuration described in "Jean-Francois Arcand's Blog" but we didn't see much missing piece.
- try to recreate the issue by running stress tests in the mirror environment with 100 users but so far no luck.
Environment:
Sun GlassFish Enterprise Server v2.1
M3000 SPARC64 VII 16GB
Solaris 10
If you have any insights for what may caused GF to not call 'close' on open sockets, we'd appreciate it!!!!
[Message sent by forum member 'ur_afroinu' (ur_afroinu)]
http://forums.java.net/jive/thread.jspa?messageID=340555