Hi,
We are experiencing problems with our GlassFish 3.0.1 production installation
when it is under high load (up to 800 concurrent users).
We have 16 tcp acceptor threads and a thread pool size of 32 on a 16
processor CPU.
A thread dump when the system is experienced to be very slow reports the
following:
glassfish_at_payback2x:/opt/glassfish/logs/trouble-shooting/20110427-091022>
grep "Thread \"http-thread-pool-8580" jvm-report-1.log Thread
"http-thread-pool-8580-(744)" thread-id: 4,581 thread-state: RUNNABLE Thread
"http-thread-pool-8580-(743)" thread-id: 4,580 thread-state: RUNNABLE Thread
"http-thread-pool-8580-(742)" thread-id: 4,579 thread-state: RUNNABLE Thread
"http-thread-pool-8580-(741)" thread-id: 4,578 thread-state: BLOCKED Waiting
on lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(740)"
thread-id: 4,577 thread-state: BLOCKED Waiting on lock:
java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(739)" thread-id: 4,576
thread-state: RUNNABLE Thread "http-thread-pool-8580-(738)" thread-id: 4,575
thread-state: RUNNABLE Thread "http-thread-pool-8580-(737)" thread-id: 4,574
thread-state: RUNNABLE Thread "http-thread-pool-8580-(736)" thread-id: 4,573
thread-state: RUNNABLE Thread "http-thread-pool-8580-(733)" thread-id: 4,556
thread-state: RUNNABLE Thread "http-thread-pool-8580-(732)" thread-id: 4,554
thread-state: BLOCKED Waiting on lock: java.util.Vector_at_faf68b Thread
"http-thread-pool-8580-(731)" thread-id: 4,547 thread-state: BLOCKED Waiting
on lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(730)"
thread-id: 4,546 thread-state: BLOCKED Waiting on lock:
java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(727)" thread-id: 4,543
thread-state: RUNNABLE Thread "http-thread-pool-8580-(726)" thread-id: 4,542
thread-state: BLOCKED Waiting on lock: java.util.Vector_at_faf68b Thread
"http-thread-pool-8580-(725)" thread-id: 4,541 thread-state: BLOCKED Waiting
on lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(724)"
thread-id: 4,540 thread-state: RUNNABLE Thread "http-thread-pool-8580-(722)"
thread-id: 4,537 thread-state: BLOCKED Waiting on lock:
java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(721)" thread-id: 4,536
thread-state: BLOCKED Waiting on lock: java.util.Vector_at_faf68b Thread
"http-thread-pool-8580-(719)" thread-id: 4,522 thread-state: BLOCKED Waiting
on lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(716)"
thread-id: 4,517 thread-state: RUNNABLE Thread "http-thread-pool-8580-(714)"
thread-id: 4,514 thread-state: RUNNABLE Thread "http-thread-pool-8580-(713)"
thread-id: 4,511 thread-state: BLOCKED Waiting on lock:
java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(708)" thread-id: 4,483
thread-state: BLOCKED Waiting on lock: java.util.Vector_at_faf68b Thread
"http-thread-pool-8580-(700)" thread-id: 4,475 thread-state: BLOCKED Waiting
on lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(699)"
thread-id: 4,474 thread-state: RUNNABLE Thread "http-thread-pool-8580-(691)"
thread-id: 4,440 thread-state: RUNNABLE Thread "http-thread-pool-8580-(678)"
thread-id: 4,421 thread-state: BLOCKED Waiting on lock:
java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(677)" thread-id: 4,420
thread-state: RUNNABLE Thread "http-thread-pool-8580-(673)" thread-id: 4,412
thread-state: BLOCKED Waiting on lock: java.util.Vector_at_faf68b Thread
"http-thread-pool-8580-(2)" thread-id: 185 thread-state: BLOCKED Waiting on
lock: java.util.Vector_at_faf68b Thread "http-thread-pool-8580-(1)" thread-id:
177 thread-state: RUNNABLE The blocked threads are all waiting for thread 733
which also is in the middle of a Vector.writeObject.
Can anyone explain to me what is going on?
Thread "http-thread-pool-8580-(741)" thread-id: 4,578 thread-state: BLOCKED
Waiting on lock: java.util.Vector_at_faf68b at:
java.util.Vector.writeObject(Vector.java:1012) at:
sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) at:
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at: java.lang.reflect.Method.invoke(Method.java:597) at:
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at:
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461) at:
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at: java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) at:
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at: java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at:
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at: java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) at:
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) at:
com.sun.enterprise.naming.util.NamingUtilsImpl.makeCopyOfObject(NamingUtilsImpl.java:86)
at:
com.sun.enterprise.naming.impl.LocalSerialContextProviderImpl.lookup(LocalSerialContextProviderImpl.java:134)
at:
com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:430)
at: javax.naming.InitialContext.lookup(InitialContext.java:392) at:
javax.naming.InitialContext.lookup(InitialContext.java:392)
The blocked threads are all waiting for thread 733 which also is in the
middle of a Vector.writeObject.
Can anyone explain to me what is going on?
Thread "http-thread-pool-8580-(733)" thread-id: 4,556 thread-state: RUNNABLE
at: java.util.Vector.writeObject(Vector.java:1012) at:
sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) at:
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at: java.lang.reflect.Method.invoke(Method.java:597) at:
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at:
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461) at:
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at: java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) at:
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at: java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at:
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at: java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) at:
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) at:
com.sun.enterprise.naming.util.NamingUtilsImpl.makeCopyOfObject(NamingUtilsImpl.java:86)
at:
com.sun.enterprise.naming.impl.LocalSerialContextProviderImpl.lookup(LocalSerialContextProviderImpl.java:134)
at:
com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:430)
at: javax.naming.InitialContext.lookup(InitialContext.java:392) at:
javax.naming.InitialContext.lookup(InitialContext.java:392)
--
[Message sent by forum member 'mphaua']
View Post: http://forums.java.net/node/797795