Some observations/tips for anyone running Glassfish 3.1.1 on Windows Server
2008 in a clustered environment... You may run into these like I did?
Let's assume that you have remote and local instances running.
1. You may see a strange discrepancy between "get-health" and
"list-instances". The former will correctly report that the remote instances
are running, whereas the latter will only list the local instances (if any)
as running, whereas the remote instances will be reported as stopped.
2. The command "start-instance" for a remote instance will work... but it
will fail at the end to know that it worked. The instance will start, GMS
messages will be exchanged... then the local DAS will say that it is adding a
new instance... and that's it. It will time out after a long time (10
minutes) saying the command failed. But it didn't really: get-health will
report that all is good.
3. You may see exceptions from Grizzly saying that "Connection timed out: no
further information available".
For me, I turned on logging for ShoalLogger to FINEST and then the exception
came with a key message:
[#|2011-08-15T13:52:57.717+0300|FINEST|glassfish3.1.1|ShoalLogger|_ThreadID=25;_ThreadName=Thread-2;ClassName=com.sun.enterprise.mgmt.MasterNode;MethodName=send;|Failed
to send message java.io.IOException: failed to connect to
10.248.0.170:9146:228.9.13.150:4821:testcluster1:tc1in2 at
com.sun.enterprise.mgmt.transport.grizzly.GrizzlyTCPConnectorWrapper.send(GrizzlyTCPConnectorWrapper.java:131)
at
com.sun.enterprise.mgmt.transport.grizzly.GrizzlyTCPConnectorWrapper.doSend(GrizzlyTCPConnectorWrapper.java:95)
at
com.sun.enterprise.mgmt.transport.AbstractMessageSender.send(AbstractMessageSender.java:74)
at
com.sun.enterprise.mgmt.transport.grizzly.GrizzlyNetworkManager.send(GrizzlyNetworkManager.java:621)
at com.sun.enterprise.mgmt.MasterNode.send(MasterNode.java:1294) at
com.sun.enterprise.mgmt.MasterNode.probeNode(MasterNode.java:987) at
com.sun.enterprise.mgmt.HealthMonitor.process(HealthMonitor.java:578) at
com.sun.enterprise.mgmt.HealthMonitor.receiveMessageEvent(HealthMonitor.java:311)
at
com.sun.enterprise.mgmt.transport.AbstractNetworkManager.receiveMessage(AbstractNetworkManager.java:139)
at
com.sun.enterprise.mgmt.transport.BlockingIOMulticastSender$MessageProcessTask.run(BlockingIOMulticastSender.java:361)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.net.ConnectException: Connection timed out: no further information at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at
com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.java:297)
at
com.sun.grizzly.connectioncache.client.CacheableConnectorHandler.finishConnect(CacheableConnectorHandler.java:230)
at
com.sun.enterprise.mgmt.transport.grizzly.GrizzlyTCPConnectorWrapper$CloseControlCallbackHandler.onConnect(GrizzlyTCPConnectorWrapper.java:184)
at
com.sun.grizzly.CallbackHandlerContextTask.doCall(CallbackHandlerContextTask.java:70)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
... 1 more
And the solution is very simple in retrospect, just frustrating that it is
not mentioned anywhere in the docs, and that "get-health" and
"list-instances/start-instance" seem to operate differently somehow...
Make sure that the clustered servers drop the firewall between them for all
inbound connections. In other words, create an inbound rule in Windows
Firewall that allows all connections from all other members of the cluster.
That's it: everything was smooth sailing after that for me.
--
[Message sent by forum member 'bland999']
View Post: http://forums.java.net/node/834170