Hi
I have run some heavy traffic on our application. The application has servlet endpoints that communicate with a backend through SLSBs. The war and ejb is located in different ears.
I use thread dumps to find heavy thread contention (maybe 30 threads competing for the same monitor) on a static com.sun.corba.ee.impl.oa.poa.POAManagerImpl object. This object is used by many different threads trying to access different EJBs. The threads seem to block on enter() and exit() methods.
I do not get why the POAManager is (seemingly) shared by all EJBs in the server?
Thanks,
-Kristoffer
3 thread examples showing this thread contention:
"httpSSLWorkerThread-8080-1" daemon prio=10 tid=0x00002aabb88af400 nid=0x5bfc waiting for monitor entry [0x00000000484ab000..0x00000000484acd20]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.sun.corba.ee.impl.oa.poa.POAManagerImpl.exit(POAManagerImpl.java:590)
- locked <0x00002aaabc74b140> (a com.sun.corba.ee.impl.oa.poa.POAManagerImpl)
at com.sun.corba.ee.impl.oa.poa.POAImpl.exit(POAImpl.java:1636)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.runServantPostInvoke(CorbaMessageMediatorImpl.java:2418)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponseHelper(CorbaMessageMediatorImpl.java:2344)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponseHelper(CorbaMessageMediatorImpl.java:2320)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponse(CorbaMessageMediatorImpl.java:2165)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createReply(CorbaMessageMediatorImpl.java:718)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:156)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:687)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:227)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1846)
at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:183)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.foobar.foobar.foobar.config.dm.__ConfigDmRemoteIf_Remote_DynamicStub.fetchNode(com/foobar/foobar/foobar/config/dm/__ConfigDmRemoteIf_Remote_DynamicStub.java)
at com.foobar.foobar.foobar.config.dm._ConfigDmRemoteIf_Wrapper.fetchNode(com/foobar/foobar/foobar/config/dm/_ConfigDmRemoteIf_Wrapper.java)
---
"httpSSLWorkerThread-8080-127" daemon prio=10 tid=0x00002aabd4b82000 nid=0x5fe8 waiting for monitor entry [0x000000006769b000..0x000000006769cd20]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.sun.corba.ee.impl.oa.poa.POAManagerImpl.enter(POAManagerImpl.java:566)
- waiting to lock <0x00002aaabc74b140> (a com.sun.corba.ee.impl.oa.poa.POAManagerImpl)
at com.sun.corba.ee.impl.oa.poa.POAImpl.enter(POAImpl.java:1576)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.getServantWithPI(CorbaServerRequestDispatcherImpl.java:378)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:224)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1846)
at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:183)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.foobar.foobar.foobar.subscriber.__SubscriberDataRetrieverRemote_Remote_DynamicStub.getCompleteUser(com/foobar/foobar/foobar/subscriber/__SubscriberDataRetrieverRemote_Remote_DynamicStub.java)
at com.foobar.foobar.foobar.subscriber._SubscriberDataRetrieverRemote_Wrapper.getCompleteUser(com/foobar/foobar/foobar/subscriber/_SubscriberDataRetrieverRemote_Wrapper.java)
at com.foobar.foobar.portal.foobar.account.AccountListingBean.isLoginValid(AccountListingBean.java:311)
---
"httpSSLWorkerThread-8080-124" daemon prio=10 tid=0x00002aabd531bc00 nid=0x5fe5 waiting for monitor entry [0x0000000067398000..0x0000000067399aa0]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.sun.corba.ee.impl.oa.poa.POAManagerImpl.exit(POAManagerImpl.java:590)
- waiting to lock <0x00002aaabc74b140> (a com.sun.corba.ee.impl.oa.poa.POAManagerImpl)
at com.sun.corba.ee.impl.oa.poa.POAImpl.exit(POAImpl.java:1636)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.runServantPostInvoke(CorbaMessageMediatorImpl.java:2418)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponseHelper(CorbaMessageMediatorImpl.java:2344)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponseHelper(CorbaMessageMediatorImpl.java:2320)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createResponse(CorbaMessageMediatorImpl.java:2165)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.createReply(CorbaMessageMediatorImpl.java:718)
at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:156)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:687)
at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:227)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1846)
at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:183)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.foobar.foobar.foobar.accountidentity.__AccountIdentityRemote_Remote_DynamicStub.getSubscriberByHttpAuthentication(com/foobar/foobar/foobar/accountidentity/__AccountIdentityRemote_Remote_DynamicStub.java)
at com.foobar.foobar.foobar.accountidentity._AccountIdentityRemote_Wrapper.getSubscriberByHttpAuthentication(com/foobar/foobar/foobar/accountidentity/_AccountIdentityRemote_Wrapper.java)
[Message sent by forum member 'ekrisjo' (kristoffer.sjogren_at_ericsson.com)]
http://forums.java.net/jive/thread.jspa?messageID=385920