users@glassfish.java.net

Re: Server stops responding due to Glassfish

From: <glassfish_at_javadesktop.org>
Date: Thu, 02 Oct 2008 08:17:12 PDT

Dear Jacob,
Thank you very much for taking the time to answer my question.
I followed your advice and run the JConsole last Sunday.

The CPU and memory values look good:
Process CPU time: 3 hours
CPU usage average: 0.4%
Heap Memory: oscillates between 0.6Gb and 1.2Gb (which means that the GC is performing OK)

The thread values are odd: starting in 100 goes to 130 in 2 hours. Stays in 130 for 3 days and goes to 175. Stays on 175 for one day and goes to 200.
The thread count never goes down and seems to keep rising.

Here it’s my thread configuration:
<http-service>
<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="10" request-timeout-in-seconds="30" thread-count="400" thread-increment="10"/>
<keep-alive max-connections="250" thread-count="1" timeout-in-seconds="30"/>
<connection-pool max-pending-count="-1" queue-size-in-bytes="-1" receive-buffer-size-in-bytes="4096" send-buffer-size-in-bytes="8192"/>
...
</http-service>

<thread-pools>
  <thread-pool idle-thread-timeout-in-seconds="120" max-thread-pool-size="400" min-thread-pool-size="0" num-work-queues="1" thread-pool-id="thread-pool-1"/>
</thread-pools>

I executed the generate-jvm-report --type=thread command and got the following result (I will not post the complete report because it may be annoying for the post’s readers):
Number of threads: 201
Number of daemon threads: 179
…..
No deadlock found

However, I did find 41 threads as follows:
Thread "httpSSLWorkerThread-443-59" thread-id 4,888 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline_at_845a92
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:485)
         at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
         at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)

and 69 threads as follows:
Thread "httpWorkerThread-80-54" thread-id 4,199 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline_at_1d6d430
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:485)
         at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
         at: com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)

Definitely, the issue I’m having is related with the threads. I checked the database connections and it seems to be closing and releasing resources OK (I’m using Connector/J 5.1 to access to a MySQL database that runs on a different server), so I can’t tell exactly what is the cause that prevents the thread count to drop and keep creating more threads. Is there any way to kill threads regardless of their state upon time out?

I would really appreciate any thought on this matter.

Thank you for your time and help,
David.
[Message sent by forum member 'zukerman' (zukerman)]

http://forums.java.net/jive/thread.jspa?messageID=302967