Hello,
I run Glassfish v2, build b04-fcs in production environment (1 CPU P4 3.4GHz,
2GB RAM, SCSI RAID 1), EJB Application + Web module. We use also JMS. All
calls to EJB are using Local interfaces.
Issue: Glassfish utilized 100% of CPU time.
In Unix TOP command (using Shift + H), I see 3 of Unix processes to utilize
33% from the CPU time. All they are JVM.
Then using {$java.home}/bin/jstack PID to get stacktrace of given overloaded
process, I see a lot of BLOCKED threads into retrieved stacktrace.
What can be the reason for that issue? Anything additional that I can send
to be able to get any kind of help or suggestion or recommendation.
Thanks to everybody in advance!
Here is a short piece from the output:
Thread 26740: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26739: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26738: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26737: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26736: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26735: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26734: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26733: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26732: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26731: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26730: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26729: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=28,
line=114 (Interpreted frame)
Thread 26728: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=146
(Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=4, line=504 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
(Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=674
(Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=11,
line=107 (Interpreted frame)
Thread 26727: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=146
(Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=4, line=504 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
(Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=674
(Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=11,
line=107 (Interpreted frame)
Thread 26726: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=146
(Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=135 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=4, line=504 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=47, line=470
(Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=674
(Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run() @bci=11,
line=107 (Interpreted frame)
Thread 26725: (state = BLOCKED)
- sun.nio.ch.PollArrayWrapper.poll0(long, int, long) @bci=0 (Compiled
frame; information may be imprecise)
- sun.nio.ch.PollArrayWrapper.poll(int, int, long) @bci=13, line=100
(Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled
frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- com.sun.enterprise.server.ss.provider.ASSelector.select(long) @bci=5,
line=133 (Compiled frame)
- com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect()
@bci=17, line=1332 (Compiled frame)
- com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener()
@bci=25, line=1279 (Compiled frame)
- com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint()
@bci=34, line=1242 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.SelectorThread.run() @bci=1,
line=1218 (Interpreted frame)
Thread 26724: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26723: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26722: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26721: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26720: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26719: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26718: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26717: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26716: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26715: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26714: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26713: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26712: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26711: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26710: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26709: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26708: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26707: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26706: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26705: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26704: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26703: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26702: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26701: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26700: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26699: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26698: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26697: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26696: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26695: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)
Thread 26694: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask()
@bci=23, line=291 (Interpreted frame)
- com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run()
@bci=28, line=104 (Interpreted frame)