users@glassfish.java.net

Two node cluster in-memory session replication not working in 2.1.1.b31g

From: <glassfish_at_javadesktop.org>
Date: Fri, 21 May 2010 14:47:18 PDT

I setup a two node cluster with identical hardware on each server and the following software:

- Ubuntu 9.10 karmic
- Glassfish v2.1.1-b31g-linux
- Java 1.6.0_20-b02
- Openntpd

I followed the regular instructions, including this tutorial: http://www.randombugs.com/java/glassfish/how-to-install-and-configure-a-glassfish-cluster.html.

The installation goes very well and the cluster start with no problems, but in-memory session replication never works for the clusterjsp sample app, I always get different session Ids when I hit each node's url.

I verified already the obvious issues such as:
- Server name resolution, all hosts can see each other, hardcoded names in each hosts file
- Mutlticast communication working properly in all nodes, verified using the shoal tests testing as sniffer and as client on each side, I do receive the nine expected messages.
- Servers are in the same subnet and also in the same switch under the same router
- OpenNTP daemon is up and running on each server to keep the clocks in synch.


Below is the relevant output I get from the server.log on the first instance that is started by the first agent. Once the agent finishes starting up replication is disabled and the app turns to memory only.

This instance is running in the same server that acts as the DAS. I also tried running the first agent in a different server and had the cluster running with no agents in the same machine. I got the same result from the remote agent, see below:


[#|2010-05-21T14:08:23.606-0700|INFO|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;/clusterjsp;replicated;web-method;session;|WEB0130: Enabling ha-based persistence for web module [/clusterjsp]'s sessions: persistence-type = [replicated] / persistenceFrequency = [web-method] / persistenceScope = [session]|#]

[#|2010-05-21T14:08:33.159-0700|WARNING|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=18;_ThreadName=Thread-46;_RequestID=ff6b9def-a602-4328-b199-bcbc0afdd96d;|ReplicationHealthChecker:replication health now ok: currentPartner: node1-instance|#]

[#|2010-05-21T14:08:33.161-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=18;_ThreadName=Thread-46;|bound pipes: 5 brokenClosedPipes:0 pipepool.size=5 total pipes:5|#]

[#|2010-05-21T14:08:39.193-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = 4|#]

[#|2010-05-21T14:08:40.196-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = 3|#]

[#|2010-05-21T14:08:41.198-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = 2|#]

[#|2010-05-21T14:08:42.200-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = 1|#]

[#|2010-05-21T14:08:43.202-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = 0|#]

[#|2010-05-21T14:08:43.203-0700|WARNING|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;_RequestID=4ed6cd02-2870-4981-8b47-2780a3d25d48;|Out of pipes in JxtaReplicationSender pipepool. Disabling replication.|#]

[#|2010-05-21T14:08:43.207-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|beginning attempt to reconnect|#]

[#|2010-05-21T14:08:49.214-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|attempt to reconnect succeeded|#]

[#|2010-05-21T14:08:50.216-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|DIAGNOSTIC Message: JxtaReplicationSender.putPipeWrapper => about to call decrementPipePoolCount because thePipeWrapper is null|#]

[#|2010-05-21T14:08:50.217-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|JxtaSenderPipeManager::pipePoolCount decremented = -1|#]

[#|2010-05-21T14:08:50.383-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|com.sun.enterprise.ee.web.sessmgmt.JxtaSocketChannel created server socket successfully|#]

[#|2010-05-21T14:08:50.394-0700|INFO|sun-appserver2.1|com.sun.enterprise.ee.web.sessmgmt|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;|PlainSocketChannel created server socket at /0.0.0.0:9999, PublishAddress = {LISTENPORT=9999, HOSTIP=192.168.18.107}|#]

[#|2010-05-21T14:08:50.398-0700|WARNING|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;_RequestID=4ed6cd02-2870-4981-8b47-2780a3d25d48;|Default Removal size threshold: 1|#]

[#|2010-05-21T14:08:50.405-0700|WARNING|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=RMI TCP Connection(7)-192.168.18.107;_RequestID=4ed6cd02-2870-4981-8b47-2780a3d25d48;|Default Removal interval threshold: 15000|#]


Any help or additional pointer is greatly appreciated.

Thanks.
[Message sent by forum member 'chinesco']

http://forums.java.net/jive/thread.jspa?messageID=470953