Hi All,
I have an urgent request for assistance. After a year and a half of development we're finally moving into product, but we have a crazy production bug.
We have a couple of seperate instances under management, one for the web tier and one for the ejb tier. On the web tier we have a couple of timers (java.util.Timers) that are started by context listeners, that every 20 minutes call an EJB to see if reports need to be generated and made available to the web users. Because of some bugs in glassfish with injecting ejbs in this scenario, we look them up using a remote initial context each time the timer runs...which works fine for about an hour and a half, then everything goes crazy. Every call to an ejb gives a time out, and our entire application stops working. There is nothing of note in the ejb server logs that I can see (definitely no errors anyway), but in the web tier logs we get the following:
[#|2009-06-22T19:28:09.398+1000|FINE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;ClassName=com.megajobindex.web.scheduling.ReportGenerationMonitor;MethodName=run;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|attempting to load unsecured operation bean from jndi ejb/UnsecuredOperationsBean|#]
[#|2009-06-22T20:17:39.393+1000|WARNING|sun-appserver2.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=24;_ThreadName=Timer-3;1800000;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|"IOP00410219: (COMM_FAILURE) Communications timeout waiting for response. Exceeded 1,800,000 milliseconds"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
|#]
[#|2009-06-22T20:17:39.400+1000|SEVERE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|Could not determine if reports need to be generated
javax.naming.NamingException: ejb ref resolution error for remote business interfacecom.megajobindex.ejb.UnsecuredOperationsRemote [Root exception is java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe]
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:425)
at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
... 7 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
... 15 more
|#]
Can anyone shed some light onto why this may be ocurring...I'm hoping it's just a configuration issue.
Cheers
Adam
Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how:
http://au.mobile.yahoo.com/mail