I found that in some cases failover is working the problem is that when the client looses a connection to one instance the following I'm receiving the following error:
-- EJB client application log
[#|2009-06-05T11:39:36.959-0300|WARNING|sun-appserver9.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=18;_ThreadName=httpSSLWorkerThread-8080-0;IIOP_CLEAR_TEXT;172.30.1.215;3150;_RequestID=0d1477b4-4e6d-4dfc-b0b3-0d0e29b5b4d8;|"IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 172.30.1.215; port: 3150"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2690)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2711)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:261)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:274)
at com.sun.corba.ee.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:130)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:192)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:181)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:186)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at pe.gob.bcrp.segur.ejb.remote.__SeguridadServiceEJBRemote_Remote_DynamicStub.login(pe/gob/bcrp/segur/ejb/remote/__SeguridadServiceEJBRemote_Remote_DynamicStub.java)
at pe.gob.bcrp.segur.ejb.remote._SeguridadServiceEJBRemote_Wrapper.login(pe/gob/bcrp/segur/ejb/remote/_SeguridadServiceEJBRemote_Wrapper.java)
at bcrp.ServletSUNWEBXML.doPost(ServletSUNWEBXML.java:110)
at bcrp.ServletSUNWEBXML.doGet(ServletSUNWEBXML.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection refused
at com.sun.enterprise.iiop.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:347)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:244)
... 40 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at com.sun.corba.ee.impl.orbutil.ORBUtility.openSocketChannel(ORBUtility.java:105)
at com.sun.enterprise.iiop.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:332)
... 41 more
|#]
And after several "Connection Refused" errors I'm seeing:
[#|2009-06-05T11:44:50.655-0300|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=httpSSLWorkerThread-8080-1;_RequestID=60461a7d-15a6-4850-8901-c95c4b6262c4;|
javax.ejb.EJBException: nested exception is: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at pe.gob.bcrp.segur.ejb.remote.__SeguridadServiceEJBRemote_Remote_DynamicStub.login(pe/gob/bcrp/segur/ejb/remote/__SeguridadServiceEJBRemote_Remote_DynamicStub.java)
at pe.gob.bcrp.segur.ejb.remote._SeguridadServiceEJBRemote_Wrapper.login(pe/gob/bcrp/segur/ejb/remote/_SeguridadServiceEJBRemote_Wrapper.java)
at bcrp.ServletSUNWEBXML.doPost(ServletSUNWEBXML.java:110)
at bcrp.ServletSUNWEBXML.doGet(ServletSUNWEBXML.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2862)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2880)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1788)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1263)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2946)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2965)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:2000)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1713)
... 2 more
Caused by: java.io.IOException: End-of-stream
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1989)
... 3 more
javax.ejb.EJBException: nested exception is: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at pe.gob.bcrp.segur.ejb.remote._SeguridadServiceEJBRemote_Wrapper.login(pe/gob/bcrp/segur/ejb/remote/_SeguridadServiceEJBRemote_Wrapper.java)
at bcrp.ServletSUNWEBXML.doPost(ServletSUNWEBXML.java:110)
at bcrp.ServletSUNWEBXML.doGet(ServletSUNWEBXML.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
|#]
On the application server holding the EJBs the error log is:
[#|2009-06-05T09:44:53.747-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.corba.ee._CORBA_.rpc.transport|_ThreadID=15;_ThreadName=p: thread-pool-1;
w: 5;Thread[p: thread-pool-1; w: 5,5,main];_RequestID=ae63c9f9-d5bc-496c-996d-9a9a7141219f;|"IOP00710311: (INTERNAL) Worker thread Thread[p: thread-pool-1; w
: 5,5,main] caught throwable while executing work."
org.omg.CORBA.INTERNAL: vmcid: SUN minor code: 311 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7703)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7722)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:558)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2721)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2743)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.writeLock(SocketOrChannelConnectionImpl.java:1065)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.close(SocketOrChannelConnectionImpl.java:804)
at com.sun.corba.ee.impl.transport.ReaderThreadImpl.doWork(ReaderThreadImpl.java:124)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
|#]
[#|2009-06-05T09:44:54.132-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.corba.ee._CORBA_.rpc.transport|_ThreadID=15;_ThreadName=p: thread-pool-1;
w: 5;Thread[p: thread-pool-1; w: 5,5,main];_RequestID=ae63c9f9-d5bc-496c-996d-9a9a7141219f;|"IOP00710311: (INTERNAL) Worker thread Thread[p: thread-pool-1; w
: 5,5,main] caught throwable while executing work."
org.omg.CORBA.INTERNAL: vmcid: SUN minor code: 311 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7703)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7722)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:558)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2721)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2743)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.writeLock(SocketOrChannelConnectionImpl.java:1065)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.close(SocketOrChannelConnectionImpl.java:804)
at com.sun.corba.ee.impl.transport.ReaderThreadImpl.doWork(ReaderThreadImpl.java:124)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
|#]
I saw there is a bug (
http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=b0e42889d787d43dd770e4deae8e?bug_id=2152079) that can be related to this but the release of the jdk I'm using should have that bug fixed.
[Message sent by forum member 'mcapurro' (mcapurro)]
http://forums.java.net/jive/thread.jspa?messageID=349427