users@glassfish.java.net

Problem setting up 2 machine cluster

From: <glassfish_at_javadesktop.org>
Date: Mon, 05 Nov 2007 10:10:18 PST

Here's the setup:

Two machines with a base install of Fedora 7, glassfish1 and glassfish2. What I've been trying to do is set up a two machine cluster with the DAS on glassfish1. I've follwed the blogs at http://blogs.sun.com/technical/entry/2_machine_cluster_setup_cli, http://blogs.sun.com/bloggerkedar/entry/a_recipe_to_create_clusters, and http://blogs.sun.com/anilam/entry/glassfish_v2_admin_console_series all without success.

I began as such:

Install Glassfish and create a domain with DAS on glassfish1
    lib/ant/bin/ant -f setup-cluster.xml
    asadmin start-domain domain1
    asadmin create-node-agent --host localhost --port 4848 g1-a1
    asadmin create-cluster --host localhost --port 4848 cluster1
    asadmin create-instance --host localhost --port 4848 --nodeagent g1-a1 --cluster cluster1 glassfish1-instance1
    asadmin start-node-agent --syncinstances=true g1-a1

That worked fine.

Then on to glassfish2 to create a node-agent:
    lib/ant/bin/ant -f setup-cluster.xml create-node-agent -Ddas.host=glassfish1
    asadmin create-instance --host glassfish1 --port 4848 --nodeagent na --cluster cluster1 glassfish2-instance1
    asadmin start-node-agent --syncinstances=true na

But that last line failed with this error:

[root_at_glassfish2 glassfish]# asadmin start-node-agent g2-a1
Please enter the admin user name>*****
Please enter the admin password>******
Please enter the master password [Enter to accept the default]:>
Redirecting output to /opt/glassfish/nodeagents/g2-a1/agent/logs/server.log
Redirecting application output to /opt/glassfish/nodeagents/g2-a1/agent/logs/server.log
Node Agent g2-a1 failed to startup. Please check the server log for more details.
CLI137 Command start-node-agent failed.

Checking the server logs in /opt/glassfish/nodeagents/g2-a1/agent/server.log I find this:

Nov 5, 2007 12:19:03 PM com.sun.enterprise.admin.servermgmt.launch.ASLauncher buildCommand
INFO:
/usr/lib/jvm/java-1.5.0-sun-1.5.0.13/jre/../bin/java
-Dcom.sun.aas.instanceRoot=/opt/glassfish/nodeagents/g2-a1/agent
-Dcom.sun.aas.configRoot=/opt/glassfish/config
-Dcom.sun.aas.defaultLogFile=/opt/glassfish/nodeagents/g2-a1/agent/logs/server.log
-Dcom.sun.aas.instanceName=g2-a1
-Dcom.sun.aas.isNodeAgent=true
-Dcom.sun.aas.promptForIdentity=true
-Dcom.sun.appserv.admin.pluggable.features=com.sun.enterprise.ee.admin.pluggable.EEClientPluggableFeatureImpl
-Dcom.sun.appserv.nss.db=/opt/glassfish/nodeagents/g2-a1/agent/config
-Dcom.sun.appserv.pluggable.features=com.sun.enterprise.ee.server.pluggable.EEPluggableFeatureImpl
-Djava.endorsed.dirs=/opt/glassfish/lib/endorsed
-Djava.library.path=/opt/glassfish/lib:/opt/glassfish/lib:/opt/glassfish/lib
-Djava.security.auth.login.config=/opt/glassfish/nodeagents/g2-a1/agent/config/login.conf
-Djava.util.logging.manager=com.sun.enterprise.server.logging.ServerLogManager
-Djmx.invoke.getters=true
-XX:+UnlockDiagnosticVMOptions
-XX:LogFile=/opt/glassfish/nodeagents/g2-a1/agent/logs/jvm.log
-XX:+LogVMOutput
-cp
/opt/glassfish/lib/appserv-launch.jar:/opt/glassfish/lib/appserv-se.jar:/opt/glassfish/lib/appserv-admin.jar:/opt/glassfish/li
b/appserv-ext.jar:/opt/glassfish/lib/appserv-rt.jar:/opt/glassfish/lib/javaee.jar:/opt/glassfish/lib/appserv-ee.jar
com.sun.enterprise.ee.nodeagent.NodeAgentMain
start
 startInstancesOverride=NOTSET
syncInstances=false
[#|2007-11-05T12:19:03.624-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|Iden
tityManager Data: User:admin|#]

[#|2007-11-05T12:19:03.728-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
0004:Starting Node Agent...|#]

[#|2007-11-05T12:19:03.743-0500|CONFIG|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|DA
S url = service:jmx:rmi:///jndi/rmi://glassfish1:8686/management/rmi-jmx-connector|#]

[#|2007-11-05T12:19:03.831-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
0025:The node agent could not configure logging levels. Default logging level will be used.|#]

[#|2007-11-05T12:19:03.832-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NAGT
0038:Executing Synchronization for node-agent With DAS|#]

[#|2007-11-05T12:19:12.920-0500|INFO|sun-appserver9.1|javax.ee.enterprise.system.tools.synchronization|_ThreadID=10;_ThreadNam
e=main;|SYNC001: Unable to communicate with Domain Administration Server. Skipping synchronization.|#]

[#|2007-11-05T12:19:12.922-0500|SEVERE|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|NA
GT0035:The NodeAgent failed to complete the intial synchronization with the DAS. Please make sure the DAS is running and is a
ccessible from the NodeAgents server|#]

[#|2007-11-05T12:19:14.925-0500|WARNING|sun-appserver9.1|javax.ee.enterprise.system.nodeagent|_ThreadID=10;_ThreadName=main;|N
AGT0013:Stopping Node Agent...|#]


So, then I deleted the node-agents, cluster and instances and started over, this time following a different blog where I created a node-agent on glassfish1 (OK), started that node-agent (OK), created a node agent on glassfish2 (OK), and tried to start that node-agent (FAILED with the same error and log as before).

At first, I had communication problems until I edited iptables to allow the ports glassfish required. As a blanket policy (and I'm not sure entirely how safe this is, but I needed to rule out blocked ports as a problem) I set glassfish1 to accept all input with a source of glassfish2 and did the reciprocal on glassfish2.

I thought maybe the problem was related to not having installed the DAS on glassfish2, even though I've read that that is not required. So I reinstalled glassfish on glassfish2 just by running the "setup-cluster.xml" normally without the "-Ddas.host=glassfish1" option I had used the first time. That didn't change anything, the same error messages still occurred, and now I have a DAS on glassfish2, which I didn't want...

Also, in the process of this, I've accumulated a few differently named folders in glassfish/nodeagents/ despite having deleted those node-agents through the admin console.

Finally, I thought I'd completely uninstall glassfish from both servers and try again fresh... but I can't seem to find any instructions on uninstalling glassfish, and I'm not about to start just deleting folders without knowing the consequences. I did find one guide for uninstalling that said to run glassfish/uninstall ... but there is no 'uninstall' in my glassfish on either server, and trying glassfish/bin/uninstall just threw this error at me:

bin/uninstall: line 10: /opt/glassfish/uninstall: No such file or directory

 

So, 4 questions:

1) How do I resolve the problem of not being able to start the node-agent on glassfish2?
2) Is it possible / How do I remove the DAS from glassfish2?
3) Is it safe to delete the folders for previously deleted node-agents from glassfish/nodeagents/?
4) Were I to uninstall and start fresh, how would I do so?

Thanks, I'll be at this all day... and I've been at it all weekend...
[Message sent by forum member 'rwillie6' (rwillie6)]

http://forums.java.net/jive/thread.jspa?messageID=243916