users@glassfish.java.net

EJB failover doesn't work in case of network disconnection

From: <glassfish_at_javadesktop.org>
Date: Thu, 18 Sep 2008 05:36:06 PDT

Hi,
for my company I test the GlassFish v2U2 application server.

We need a high available application server for JMS and EJB (with session failover), but we have no need for e.g. the webserver.

But I have a problem with failover (EJB) when I disconnect the network from one server.



Here is my test environment:

On a windows system, I have installed two debian server systems in VM Ware, server1 and server2.

server1:
- my DAS, has the node agent 'cluster1-node1'

server2:
- has just a node agent 'cluster1

cluster1:
- cluster1 is running on server1
- has two instances:
  ~ instance-ONE on 'cluster1-node1' (server1)
  ~ instance-TWO on 'cluster1-node2' (server2)


Now I try this tutorial, just with the difference, that I use two server instead of one:
http://blogs.sun.com/sviveka/entry/ejb_availability


I deployed the SFSB.ear, got the SFSBClient.ear (I run the SFSBClient.ear on the windows system) and everything works fine.
If I disable an instance with 'asadmin stop-instance instance-TWO', the instance failover and session failover works fine, I can see what had been entered before etc.


Now I try to simulate network problems, e.g. power failure on server2:
Both instances run, I start the SFSBClient.ear on my windows system and my attribute is added on instance-TWO.
Now I disabled the network interface (with ifconfig eth0 down or by disabling the ethernet card by VM Ware) on server2, but the failover doesn't work.
The SFSBClient.ear shows me the following error message:

Bean Lookup Successful
Add an Attribute(a)
Remove an Attribute(r)
Print Attributes(p)
Quit(q)
a
**********************************************
Enter name :abc
Enter value :123
Attribute abc added on instance: instance-TWO
**********************************************

[b] ===============~ "ifconfig eth0 down" on server2 ~================== [/b]

Add an Attribute(a)
Remove an Attribute(r)
Print Attributes(p)
Quit(q)
a
**********************************************
Enter name :new
Enter value :456
javax.ejb.EJBException: nested exception is: java.rmi.MarshalException: CORBA CO
MM_FAILURE 1398079696 Maybe; nested exception is:
        org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception
 is:
        org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
        at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
        at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
        at ha.failover.beans.sfsb.__SFSBRemote_Remote_DynamicStub.addAttribute(ha/failover/beans/sfsb/__SFSBRemote_Remote_DynamicStub.java)
        at ha.failover.beans.sfsb._SFSBRemote_Wrapper.addAttribute(ha/failover/beans/sfsb/_SFSBRemote_Wrapper.java)
        at ha.failover.client.EJBClient.addAttribute(Unknown Source)
        at ha.failover.client.EJBClient.processQuery(Unknown Source)
        at ha.failover.client.EJBClient.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.util.Utility.invokeApplicationMain(Utility.java:266)
        at com.sun.enterprise.appclient.MainWithModuleSupport.<init>(MainWithModuleSupport.java:449)
        at com.sun.enterprise.appclient.MainWithModuleSupport.<init>(MainWithModuleSupport.java:259)
        at com.sun.enterprise.appclient.Main.main(Main.java:200) Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2862)
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2880)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1788)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1263)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555) Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211 completed:No
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2946)
        at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2965)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:2000)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1713)
        ... 2 more
[b]Caused by: java.io.IOException: The connection was closed by remote host[/b]
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:25)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1987)
        ... 3 more
javax.ejb.EJBException: nested exception is: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
        org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
        at ha.failover.beans.sfsb._SFSBRemote_Wrapper.addAttribute(ha/failover/beans/sfsb/_SFSBRemote_Wrapper.java)
        at ha.failover.client.EJBClient.addAttribute(Unknown Source)
        at ha.failover.client.EJBClient.processQuery(Unknown Source)
        at ha.failover.client.EJBClient.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.util.Utility.invokeApplicationMain(Utility.java:266)
        at com.sun.enterprise.appclient.MainWithModuleSupport.<init>(MainWithModuleSupport.java:449)
        at com.sun.enterprise.appclient.MainWithModuleSupport.<init>(MainWithModuleSupport.java:259)
        at com.sun.enterprise.appclient.Main.main(Main.java:200)
null



The strange is, if I now try again to add a new attribute (or to show the existing ones), everything works fine:
[...]
        at com.sun.enterprise.appclient.Main.main(Main.java:200)
null
**********************************************
Add an Attribute(a)
Remove an Attribute(r)
Print Attributes(p)
Quit(q)
a
**********************************************
Enter name :second try
Enter value :789
Attribute second try added on instance: instance-ONE
**********************************************
Add an Attribute(a)
Remove an Attribute(r)
Print Attributes(p)
Quit(q)
p
**********************************************
Name : second try, Value: 789
Name : abc, Value: 123
**********************************************
Add an Attribute(a)
Remove an Attribute(r)
Print Attributes(p)
Quit(q)



I attached my appc.log, created by appclient -xml ... -client SFSBClient.jar, but it doesn't contain any help informations (for me).



Is this a problem of GlassFish, Windows, VM Ware, Debian or AppClient?



Would be happy for any suggestions.
[Message sent by forum member 'hannes2k' (hannes2k)]

http://forums.java.net/jive/thread.jspa?messageID=300087