users@glassfish.java.net

Re: Server stops responding due to Glassfish

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Thu, 02 Oct 2008 11:39:57 -0400

Salut,

glassfish_at_javadesktop.org wrote:
> Dear Jacob,
> Thank you very much for taking the time to answer my question.
> I followed your advice and run the JConsole last Sunday.
>
> The CPU and memory values look good:
> Process CPU time: 3 hours
> CPU usage average: 0.4%
> Heap Memory: oscillates between 0.6Gb and 1.2Gb (which means that the GC is performing OK)
>
> The thread values are odd: starting in 100 goes to 130 in 2 hours. Stays in 130 for 3 days and goes to 175. Stays on 175 for one day and goes to 200.
> The thread count never goes down and seems to keep rising.

Right. The number will grow until you reach the maximum, which you have
set to 400 (quite high IMO). Looking at the rate of thread creation, I
would guess 100 threads should be enough.


>
> Here it’s my thread configuration:
> <http-service>
> <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="10" request-timeout-in-seconds="30" thread-count="400" thread-increment="10"/>
> <keep-alive max-connections="250" thread-count="1" timeout-in-seconds="30"/>
> <connection-pool max-pending-count="-1" queue-size-in-bytes="-1" receive-buffer-size-in-bytes="4096" send-buffer-size-in-bytes="8192"/>
> ...
> </http-service>
>
> <thread-pools>
> <thread-pool idle-thread-timeout-in-seconds="120" max-thread-pool-size="400" min-thread-pool-size="0" num-work-queues="1" thread-pool-id="thread-pool-1"/>
> </thread-pools>
>

Thread-Pool is for Corba/others component, not the http runtime.

> I executed the generate-jvm-report --type=thread command and got the following result (I will not post the complete report because it may be annoying for the post’s readers):
> Number of threads: 201
> Number of daemon threads: 179
> …..
> No deadlock found
>
> However, I did find 41 threads as follows:
> Thread "httpSSLWorkerThread-443-59" thread-id 4,888 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline_at_845a92
> at: java.lang.Object.wait(Native Method)
> at: java.lang.Object.wait(Object.java:485)
> at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
> at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
>
> and 69 threads as follows:
> Thread "httpWorkerThread-80-54" thread-id 4,199 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline_at_1d6d430
> at: java.lang.Object.wait(Native Method)
> at: java.lang.Object.wait(Object.java:485)
> at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
> at: com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)
>
> Definitely, the issue I’m having is related with the threads.

The above trace is expected. It just means Threads are waiting for work
and are idle.

I checked the database connections and it seems to be closing and
releasing resources OK (I’m using Connector/J 5.1 to access to a MySQL
database that runs on a different server), so I can’t tell exactly what
is the cause that prevents the thread count to drop and keep creating
more threads. Is there any way to kill threads regardless of their state
upon time out?
>

No. But I doubt is is related. Can you try with 100 threads and see how
it goes? When you see the issue, do the following:

% netstat -an| grep <http-port> > fd.txt
% jstack PID > stack.txt

Let see what we can find from the two files.

A+

- Jeanfrancois



> I would really appreciate any thought on this matter.
>
> Thank you for your time and help,
> David.
> [Message sent by forum member 'zukerman' (zukerman)]
>
> http://forums.java.net/jive/thread.jspa?messageID=302967
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>