users@glassfish.java.net

Surviving back end server downtime

From: Vetle Roeim <vetler_at_gmail.com>
Date: Tue, 11 Jan 2011 10:58:02 +0100

Hi,

If an application running on server 1 tries to access EJBs remotely on
server 2 while server 2 is down, the application running on server 1 does
not seem to recover after server 2 has been started again. Is there any way
to fix this without restarting server 1? (Both servers running Glassfish
3.0.1)

In our scenario, we have a JSF application on a front end server accessing
EJBs remotely on a back end server. If we restart the back end, *without*
using the front end application when the server is stopped, everything works
fine (the application can access the EJBs after the back end server has been
started again). However, if the application is used while the back end
server is stopped, the application does not reconnect to the EJBs running on
the back end server when it is up and running again, and we have to restart
the front end server.

When using the application while the back end server is down, we get a long
stack trace ending in

Caused by: java.net.ConnectException: Connection refused: connect
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at com.sun.corba.ee.impl.orbutil.ORBUtility.openSocketChannel(ORBUtility.java:106)
        at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:325)
        ... 139 more

Which makes sense, since it obviously can't connect to the server.

When using the application again after the back end server is up and running
again, we get the following error:

Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696
Maybe; nested exception is:
        org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
        at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
        at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
        at no.evote.service.__ElectionEventService_Remote_DynamicStub.findByPk(no/evote/service/__ElectionEventService_Remote_DynamicStub.java)
        ... 130 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208
completed: Maybe
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3603)
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3621)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1726)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1216)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:492)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211
completed: No
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3687)
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3706)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1938)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1651)
        ... 3 more
Caused by: java.io.IOException: End-of-stream
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1927)
        ... 4 more

Shouldn't the application be able to reconnect to the back end server in
this case? Is there any way to accomplish this without restarting the front
end server?

Regards,
Vetle