users@glassfish.java.net

Re: too much blocked threads, Glassfish v2, build b04-fcs

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 17 Oct 2007 10:16:35 -0400

Hi,

Hilmi Hilmiev wrote:
> Hello,
>
> I run Glassfish v2, build b04-fcs in production environment (1 CPU P4
> 3.4GHz, 2GB RAM, SCSI RAID 1), EJB Application + Web module. We use also
> JMS. All calls to EJB are using Local interfaces.
>
> Issue: Glassfish utilized 100% of CPU time.
>
> In Unix TOP command (using Shift + H), I see 3 of Unix processes to
> utilize 33% from the CPU time. All they are JVM.
>
> Then using {$java.home}/bin/jstack PID to get stacktrace of given
> overloaded process, I see a lot of BLOCKED threads into retrieved
> stacktrace.
>
> What can be the reason for that issue? Anything additional that I can
> send to be able to get any kind of help or suggestion or recommendation.
>
> Thanks to everybody in advance!
>
> Here is a short piece from the output:
>
> Thread 26740: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26739: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26738: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26737: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26736: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26735: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26734: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26733: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26732: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26731: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26730: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run()
> @bci=28, line=114 (Interpreted frame)
>
>
> Thread 26729: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=28, line=114 (Interpreted frame)


All of those above are expected. This the the http thread pool waiting
for requests to execute.

>
>
> Thread 26728: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos (long) @bci=11,
> line=146 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=504 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18,
> line=674 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=11, line=107 (Interpreted frame)
>
>
> Thread 26727: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos (long) @bci=11,
> line=146 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=504 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18,
> line=674 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=11, line=107 (Interpreted frame)
>
>
> Thread 26726: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos (long) @bci=11,
> line=146 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=504 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18,
> line=674 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run ()
> @bci=11, line=107 (Interpreted frame)
>

Those above looks suspicious. Grizzly isn't using java.util.concurrent
thread pool by default (little slow ;-)), so I don't know where this
stack comes from. have you customized the thread pool in Grizzly?


The below one is expected.

>
> Thread 26725: (state = BLOCKED)
> - sun.nio.ch.PollArrayWrapper.poll0(long, int, long) @bci=0 (Compiled
> frame; information may be imprecise)
> - sun.nio.ch.PollArrayWrapper.poll (int, int, long) @bci=13, line=100
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> - com.sun.enterprise.server.ss.provider.ASSelector.select(long) @bci=5,
> line=133 (Compiled frame)
> - com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect()
> @bci=17, line=1332 (Compiled frame)
> - com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener
> () @bci=25, line=1279 (Compiled frame)
> -
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint()
> @bci=34, line=1242 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.SelectorThread.run ()
> @bci=1, line=1218 (Interpreted frame)
>
>
> Thread 26724: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26723: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26722: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26721: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26720: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26719: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26718: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26717: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26716: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26715: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26714: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26713: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26712: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26711: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26710: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26709: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26708: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26707: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26706: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26705: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26704: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26703: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26702: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26701: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26700: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26699: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26698: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26697: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26696: (state = BLOCKED)
> - java.lang.Object.wait (long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26695: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait () @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
> @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run ()
> @bci=28, line=104 (Interpreted frame)
>
>
> Thread 26694: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
> - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask
> () @bci=23, line=291 (Interpreted frame)
> - com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
> @bci=28, line=104 (Interpreted frame)


What is the value of the request-processing thread-count attribute in
domain.xml? By default is 5, but you seems to either increased the
value. Also how many http-listener do you have. Unfortunalty, the
request-processing thread-count applies to all http-listener, so if you
have x http-listener, you gonna ends up with x * thread-count blocked
thread.

I suspect your CPU spinning could be related to issue:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=3707

Is GlassFish servicing https requests?

Thanks

-- Jeanfrancois