users@glassfish.java.net

Re: Urgent - EJB SSB Timeouts!

From: Adam Jenkins <adamjenkinstmpredirect_at_yahoo.com.au>
Date: Mon, 22 Jun 2009 12:50:40 -0700 (PDT)

oh, and I'm running the following: jdk1.6.0_14 - linux 32bit glassfish v2.1 latest release (from the site 4 days ago) ubuntu 2.6.27-11-server --- On Tue, 23/6/09, Adam Jenkins <adamjenkinstmpredirect@yahoo.com.au> wrote: > From: Adam Jenkins <adamjenkinstmpredirect@yahoo.com.au> > Subject: Urgent - EJB SSB Timeouts! > To: users@glassfish.dev.java.net > Received: Tuesday, 23 June, 2009, 5:15 AM > > Hi All, > > I have an urgent request for assistance.  After a year > and a half of development we're finally moving into product, > but we have a crazy production bug. > > We have a couple of seperate instances under management, > one for the web tier and one for the ejb tier.  On the > web tier we have a couple of timers (java.util.Timers) that > are started by context listeners, that every 20 minutes call > an EJB to see if reports need to be generated and made > available to the web users.  Because of some bugs in > glassfish with injecting ejbs in this scenario, we look them > up using a remote initial context each time the timer > runs...which works fine for about an hour and a half, then > everything goes crazy.  Every call to an ejb gives a > time out, and our entire application stops working.  > There is nothing of note in the ejb server logs that I can > see (definitely no errors anyway), but in the web tier logs > we get the following: > > > > [#|2009-06-22T19:28:09.398+1000|FINE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;ClassName=com.megajobindex.web.scheduling.ReportGenerationMonitor;MethodName=run;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|attempting > to load unsecured operation bean from jndi > ejb/UnsecuredOperationsBean|#] > > [#|2009-06-22T20:17:39.393+1000|WARNING|sun-appserver2.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=24;_ThreadName=Timer-3;1800000;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|"IOP00410219: > (COMM_FAILURE) Communications timeout waiting for > response.  Exceeded 1,800,000 milliseconds" > org.omg.CORBA.COMM_FAILURE:   vmcid: > SUN  minor code: 219 completed: Maybe >     at > com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180) >     at > com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195) >     at > com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198) >     at > com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196) >     at > com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291) >     at > com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389) >     at > com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357) >     at > com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219) >     at > com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192) >     at > com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) >     at > com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225) >     at > com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java) >     at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >     at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >     at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >     at > java.lang.reflect.Method.invoke(Method.java:597) >     at > com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372) >     at > com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74) >     at > javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304) >     at > com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414) >     at > javax.naming.InitialContext.lookup(InitialContext.java:392) >     at > com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132) >     at > java.util.TimerThread.mainLoop(Timer.java:512) >     at > java.util.TimerThread.run(Timer.java:462) > |#] > > [#|2009-06-22T20:17:39.400+1000|SEVERE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|Could > not determine if reports need to be generated > javax.naming.NamingException: ejb ref resolution error for > remote business > interfacecom.megajobindex.ejb.UnsecuredOperationsRemote > [Root exception is java.rmi.MarshalException: CORBA > COMM_FAILURE 1398079707 Maybe; nested exception is: >     > org.omg.CORBA.COMM_FAILURE:   vmcid: > SUN  minor code: 219 completed: Maybe] >     at > com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:425) >     at > com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74) >     at > javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304) >     at > com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414) >     at > javax.naming.InitialContext.lookup(InitialContext.java:392) >     at > com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132) >     at > java.util.TimerThread.mainLoop(Timer.java:512) >     at > java.util.TimerThread.run(Timer.java:462) > Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE > 1398079707 Maybe; nested exception is: >     > org.omg.CORBA.COMM_FAILURE:   vmcid: > SUN  minor code: 219 completed: Maybe >     at > com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271) >     at > com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205) >     at > com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) >     at > com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225) >     at > com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java) >     at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >     at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >     at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >     at > java.lang.reflect.Method.invoke(Method.java:597) >     at > com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372) >     ... 7 more > Caused by: > org.omg.CORBA.COMM_FAILURE:   vmcid: > SUN  minor code: 219 completed: Maybe >     at > com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180) >     at > com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195) >     at > com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198) >     at > com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196) >     at > com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291) >     at > com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389) >     at > com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357) >     at > com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219) >     at > com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192) >     ... 15 more > |#] > > > Can anyone shed some light onto why this may be > ocurring...I'm hoping it's just a configuration issue. > > Cheers > Adam > > >       Access Yahoo!7 Mail on your mobile. > Anytime. Anywhere. > Show me how: http://au.mobile.yahoo.com/mail > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net > For additional commands, e-mail: users-help@glassfish.dev.java.net > > Access Yahoo!7 Mail on your mobile. Anytime. Anywhere. Show me how: http://au.mobile.yahoo.com/mail