Chapter 8 Configuring Node Agents

This chapter describes the node agents in Enterprise Server. It contains the following sections:

• What is a Node Agent?

• Server Instance Behavior After Node Agent Failure

• Deploying Node Agents

• Synchronizing Node Agents and the Domain Administration Server

• Viewing Node Agent Logs

• Working with Node Agents

What is a Node Agent?

A node agent is a lightweight process that is required on every machine that hosts server instances, including the machine that hosts the Domain Administration Server (DAS). The node agent:

• Starts, stops, creates and deletes server instances as instructed by the Domain Administration Server.

• Restarts failed server instances.

• Provides a view of the log files of failed servers.

• Synchronizes each server instance’s local configuration repository with the Domain Administration Server’s central repository. Each local repository contains only the information pertinent to that server instance or node agent.

  The following figure illustrates the overall node agent architecture:

  

When you install the Application Server, a node agent is created by default with the host name of the machine. This node agent must be manually started on the local machine before it runs.

You can create and delete server instances even if the node agent is not running. However, the node agent must be running before you use it to start and stop server instances.

A node agent services a single domain. If a machine hosts instances running in multiple domains, it must run multiple node agents.

Server Instance Behavior After Node Agent Failure

A node agent might be stopped unexpectedly, for example, by a software failure or other error. In this situation, any server instances that the node agent was managing are no longer managed. However, such server instances continue to run and remain accessible by the DAS. Information about the server instances can still be obtained through Enterprise Server administrative interfaces, and applications that are deployed on the server instances can still be accessed.

If the node agent is restarted, the server instances that are not managed remain unmamaged. The node agent does not resume the management of these server instances. If an unmanaged server instance is stopped unexpectedly, for example, by a software failure or other error, the node agent cannot restart the server instance.

If an unmanaged server instance must continue to run, you cannot resume the management of the server instance by a node agent. The only way to resume the management of an unmanaged server instance is to stop and restart the server instance after the node agent is restarted.

Deploying Node Agents

You configure and deploy node agents in two ways:

• Online deployment, when you know your topology and already have the hardware for your domain.

• Offline deployment, when you are configuring domains and server instances before setting up the full environment

To Deploy Node Agents Online

Use online deployment if you already know the domain topology and have the hardware for your domain.

The following figure summarizes the online deployment of node agents:

Before You Begin

Install and start the Domain Administration Server. Once the Domain Administration Server is up and running, begin either online or offline deployment.

1. Install a node agent on every machine that will host a server instance.

   Use the installer or the asadmin create-node-agent command . If a machine requires more than one node agent, use the asadmin create-node-agent command to create them.

   See Creating a Node Agent for more information.

2. Start the node agents using the asadmin start-node-agent command .

   When started, a node agent communicates with the Domain Administration Server (DAS). When it reaches the DAS, a configuration for the node agent is created on the DAS. Once the configuration exists, the node agent is viewable in the Admin Console.

   See Starting a Node Agent for more information.

3. Configure the domain: create server instances, create clusters, and deploying applications.

To Deploy Node Agents Offline

Use offline deployment to deploy node agents in the domain before configuring the individual local machines.

The following figure summarizes the offline deployment.

Before You Begin

Install and start the Domain Administration Server. Once the Domain Administration Server is up and running, begin either online or offline deployment.

1. Create placeholder node agents in the Domain Administration Server.

   See To Create a Node Agent Placeholder for more information.

2. Create server instances and clusters, and deploy applications.

   When creating a server instance, make sure to assign port numbers that are not already in use. Because the configuration is being done offline, the domain cannot check for port conflicts at creation time.

3. Install a node agent on every machine that will host a server instance.

   Use the installer or the asadmin create-node-agent command . The node agents must have the same names as the placeholder node agents previously created.

   See Creating a Node Agent for more information.

4. Start the node agents using the asadmin start-node-agent command .

   When a node agent is started, it binds to the Domain Administration Server and creates any server instances previously associated with the node agent.

   See Starting a Node Agent for more information.

Synchronizing Node Agents and the Domain Administration Server

Because configuration data is stored in the Domain Administration Server’s repository (the central repository) and also cached on the node agent’s local machine, the two must be synchronized. The synchronization of cache is always done on a explicit user action through the administration tools.

This section contains the following topics:

• Node Agent Synchronization

• Server Instance Synchronization

• Synchronizing Library Files

• Unique Settings and Configuration Management

• Synchronizing Large Applications

Node Agent Synchronization

When a node agent is started for the first time, it sends a request to the Domain Administration Server (DAS) for the latest information in the central repository. Once it successfully contacts the DAS and gets configuration information, the node agent is bound to that DAS.

By default, the asadmin start-node-agent command automatically starts the remote server instances without synchronizing with DAS. If you are starting a remote server instance that is synchronized with the central repository managed by DAS, specify the --startinstances=false option of the asadmin start-node-agent command. Then use the asadmin start-instance command to start the remote server instance.

If you created a placeholder node agent on the DAS, when the node agent is started for the first time it gets its configuration from the central repository of the DAS. During its initial start-up, if the node agent is unable to reach the DAS because the DAS is not running, the node agent stops and remains unbound.

If changes are made in the domain to the node agent’s configuration, they are automatically communicated to the node agent on the local machine while the node agent is running.

If you delete a node agent configuration on the DAS, the next time the node agent synchronizes, it stops and marks itself as awaiting deletion. Manually delete it using the local asadmin delete-node-agent command.

Server Instance Synchronization

If you explicitly start a server instance with the Admin Console or asadmin tool, the server instance is synchronized with the central repository. If this synchronization fails, the server instance doesn’t start.

If a node agent starts a server instance without an explicit request through the Admin Console or the asadmin tool, the repository cache for the server instance is not synchronized. The server instance runs with the configuration as stored in its cache. You must not add or remove files in the remote server instance's cache.

The remote server instance's configuration are treated as cache (all files under nodeagents/na1/server1) and owned by Application Server. In extreme cases, if user removes all files of a remote server instance and restarts the node agent, the remote server instance (for example, server1) will be recreated and all necessary files will be synchronized.

The following files and directories are kept synchronized by the Application Server.

Synchronizing Library Files

The --libraries deploy time attribute for an application can be used to specify runtime dependencies of an application. When a relative path is specified, (only the jar name), Application Server attempts to find the specified library in domain-dir/lib/applibs.

To make a library available to the whole domain, you could place the JAR file in domain-dir/lib or domain-dir/lib/classes. (For more information, see Using the Common Class Loader in Sun GlassFish Enterprise Server v2.1.1 Developer’s Guide. ) This is usually the case for JDBC drivers and other utility libraries that are shared by all applications in the domain.

For cluster-wide or stand alone server wide use, copy the jars into the domain-dir/domain1/config/xyz-config/lib directory. Next, add the jars in classpath-suffix or classpath-prefix element of xyz-config. This will synchronize the jars for all server instances using xyz-config.

In summary:

• domains/domain1/lib - domain wide scope, common class loader, adds the jars automatically.

• domains/domain1/config/cluster1, config/lib - config wide, update classpath-prefix or classpath-suffix.

• domains/domain1/lib/applibs - application scope, added to application class loader automatically

• domains/domain1/config/cluster1, config/lib/ext - adds to http://java.sun.com/j2se/1.5.0/docs/guide/extensions/extensions.html automatically.

Unique Settings and Configuration Management

Configuration files (under domains/domain1/config are synchronized across the domain. If you want to customize a server.policy file for a server1-config used by a stand alone server instance (server1), place the modified server.policy file under domains/domain1/config/server1-config directory.

This modified server.policy file will only be synchronized for the stand alone server instance, server1. You should remember to update the jvm-option. For example: <java-config> ...<jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server1-config/server.policy</jvm-options></java-config>

Synchronizing Large Applications

When your environment contains large applications to synchronize or available memory is constrained, you can adjust the JVM options to limit memory usage. This adjustment reduces the possibility of receiving out of memory errors. The instance synchronization JVM uses default settings, but you can configure JVM options to change them.

Set the JVM options using the INSTANCE-SYNC-JVM-OPTIONS property. The command to set the property is:

asadmin set 
domain.node-agent.node_agent_name.property.INSTANCE-SYNC-JVM-OPTIONS="JVM_options"

For example:

asadmin set 
domain.node-agent.node0.property.INSTANCE-SYNC-JVM-OPTIONS="-Xmx32m -Xss2m"

In this example, the node agent is node0 and the JVM options are -Xmx32m -Xss2m.

For more information, see http://java.sun.com/docs/hotspot/VMOptions.html.

Restart the node agent after changing the INSTANCE-SYNC-JVM-OPTIONS property, because the node agent is not automatically synchronized when a property is added or changed in its configuration.

Using the doNotRemoveList Flag

If your application requires to store and read files in the directories (applications, generated, docroot, config, lib, java-web-start) that are synchronized by the Application Server, use the doNotRemoveList flag. This attribute takes a coma-separated list of files or directories. Your application dependent files are not removed during server startup, even if they do not exist in the central repository managed by DAS. If the same file exists in the central repository, they will be over written during synchronization.

Use the INSTANCE-SYNC-JVM-OPTIONS property to pass in the doNotRemoveList attribute.

For example:

<node-agent name="na1" ...>

...

<property name="INSTANCE-SYNC-JVM-OPTIONS" value="-Dcom.sun.appserv.doNotRemoveList=applications/j2ee-modules/<webapp_context>/logs,generated/mylogdir"/>

</node–agent>

Setting Timeouts for Starting and Stopping Node Agents

The following properties are available in Enterprise Server for setting the timeouts for starting and stopping the node agents: com.sun.aas.startAgentTimeout and com.sun.aas.startAgentTimeout.

These properties can be added to the as-install/lib/processLauncher.xml file. These properties are defined in milliseconds and they provide graceful starting and stopping of node agents when the --startinstances option is enabled.

If the node agent name is mynodegant1, find the <process name="mynodegant1"> entry in the /processLauncher.xml file and add the following lines:

<sysproperty key="com.sun.aas.startAgentTimeout" value="1500000"/>

<sysproperty key="com.sun.aas.stopAgentTimeout" value="240000"/>

Viewing Node Agent Logs

Each node agent has its own log file. If you experience problems with a node agent, see the log file at:

node_agent_dir /node_agent_name/agent/logs/server.log .

Sometimes the node agent log instructs you to look at a server’s log to get a detailed message about a problem.

The server logs are located at:

node_agent_dir/node_agent_name/ server_name/logs/server.log

The default location for node_agent_dir is install_dir/nodeagents.

Working with Node Agents

• How to Perform Node Agent Tasks

• Node Agent Placeholders

• To Create a Node Agent Placeholder

• Creating a Node Agent

• Starting a Node Agent

• Stopping a Node Agent

• Deleting a Node Agent

• To View General Node Agent Information

• To Delete a Node Agent Configuration

• To Edit a Node Agent Configuration

• To Edit a Node Agent Realm

• To Edit the Node Agent’s Listener for JMX

• To Create a Standalone Server Instance

• Support for Installing DAS, Node Agent, and Server Instance in a DMZ

How to Perform Node Agent Tasks

Some node agent tasks require you to use the asadmintool locally on the system where the node agent runs. Other tasks you can perform remotely using either the Admin Console or asadmin.

The following table summarizes the tasks and where to run them:

Node Agent Placeholders

You can create and delete server instances without an existing node agent using a node agent placeholder. The node agent placeholder is created on the Domain Administration Server (DAS) before the node agent itself is created on the node agent’s local system.

For information on creating a node agent placeholder, see To Create a Node Agent Placeholder

Once you’ve created a placeholder node agent, use it to create instances in the domain. However, before starting the instances you must create and start the actual node agent locally on the machine where the instances will reside using the asadmin command. See Creating a Node Agent and Starting a Node Agent

To Create a Node Agent Placeholder

A node agent is a local watchdog for server instances that are running on a remote machine. Therefore, a node agent must be created on the machine that is hosting the server instances. As a result of this requirement, you can use the Admin Console to create only a placeholder for a node agent. This placeholder is a node agent configuration for which a node agent does not yet exist.

After creating a placeholder, use the asadmin command create-node-agent on the machine hosting the node agent to complete the creation. For more information, see Creating a Node Agent.

For a list of the steps involved in creating and using node agents, see Deploying Node Agents.

1. In the tree component, select the Node Agents node.

2. On the Node Agents page, click New.

3. On the Current Node Agent Placeholder page, enter a name for the new node agent.

   The name must be unique across all node agent names, server instance names, cluster names, and configuration names in the domain.

4. Click OK.

   The placeholder for your new node agent is listed on the Node Agents page.

Equivalent asadmin command

create-node-agent-config

Creating a Node Agent

To create a node agent, run the asadmin command create-node-agent locally on the machine on which the node agent runs.

The default name for a node agent is the host name on which the node agent is created.

If you’ve already created a node agent placeholder, use the same name as the node agent placeholder to create the associated node agent. If you have not created a node agent placeholder, and the DAS is up and reachable, the create-node-agent command also creates a node agent configuration (placeholder) on the DAS.

For a complete description of the command syntax, see the online help for the command.

The DAS and a node agent might be configured to communicate securely. In this situation, when the node agent is started, it must validate the certificate that the DAS sends to the node agent. To validate the certificate, the node agent looks up the certificate in the node agent's local truststore , which is protected by a master password. To enable the node agent to be started without prompting the user for a password, save the node agent's master password to a file when you create the node agent. If you do not save the node agent's master password to a file, the user is prompted for the master password whenever the user starts the node agent.

In some situations you must specify the name of a host that can be reached through DNS. For more information, see To Create a Node Agent for a DNS-Reachable Host.

To Create a Node Agent

1. Type the following command:

   asadmin create-node-agent --host das-host --port port-no --user das-user [--savemasterpassword=true] nodeagent

   To enable the node agent to be started without prompting the user for a password, save the node agent's master password to a file. To save the node agent's master password to a file, set the --savemasterpassword option to true in the command to create the node agent.

   If you set --savemasterpassword to true, you are prompted for the master password. Otherwise, you are not prompted for a password.

   --host das-host

   Specifies the name of the host where the Domain Administration Server (DAS) is running.

   -port port-no

   Specifies the HTTP/HTTPS port number for administering the domain.

   --user das-user

   Specifies the DAS user.

   nodeagent

   Specifies the name of the node agent that you are creating. This name must be unique in the domain.

Example 8–1 Creating a Node Agent

asadmin create-node-agent --host myhost --port 4848 --user admin nodeagent1

This command creates a node agent that is named nodeagent1. The DAS with which the node agent communicates is running on the machine myhost. The HTTP port for administering the agent's domain is 4848. The name of the DAS user is admin.

To Create a Node Agent for a DNS-Reachable Host

The host where the DAS is running must be reachable through DNS in the following situations:

• Domains cross subnet boundaries, that is, the node agent and the DAS are in different domains, for example, sun.com and java.com.

• A DHCP machine is being used whose host name is not registered in DNS.

1. In the create-domain command to create the domain, specify the --domainproperties domain.hostName=das-host-name option.

   das-host-name is the name of the machine where the DAS is running.

2. In the create-node-agent command to create the node agent, specify the following options:

   • --host das-host-name, where das-host-name is the DAS host name that you specified in Step 1. This option corresponds to the agent.das.host property in the file as-install/nodeagents/nodeagentname/agent/config/das.properties.

   • --agentproperties remoteclientaddress=node-agent-host-name, where node-agent-host-name is the host name that the DAS uses to connect to the node agent. This option corresponds to the agent.client.host property in the file as-install/nodeagents/nodeagentname/agent/config/nodeagent.properties.

Specifying the Host by Updating the hosts File

Another solution is to update the hosts hostname/IP resolution file specific to the platform so the hostname resolves to the correct IP address. However, when reconnecting using DHCP you might get assigned a different IP address. In that case, you must update the host resolution files on each server.

Starting a Node Agent

Before a node agent can manage server instances, it must be running. Start a node agent by running the asadmin command start-node-agent locally on the system where the node agent resides.

For a complete description of the command syntax, see the online help for the command.

For example:

asadmin start-node-agent --user admin --startinstances=false nodeagent1

where admin is your administration user, and nodeagent1 is the node agent being started.

By default, the cache repositories of node agent instances are not synchronized from the central repository when a node agent is restarted. To forcibly synchronize the instances' cache repositories with central repository, set the --syncinstances option to true in the asadmin start-node-agent command.

if you set the --syncinstances option to true, the repositories of all instances are synchronized when the node agent is restarted.

After restarting a node agent, use the asadmin start-instance command to start the server instance.

Stopping a Node Agent

Run the asadmin command stop-node-agent on the system where the node agent resides to stop a running node agent. The stop-node-agent command stops all server instances that the node agent manages.

For a complete description of the command syntax, see the online help for the command.

For example:

asadmin stop-node-agent nodeagent1

where nodeagent1 is the name of the node agent.

Deleting a Node Agent

Before deleting a node agent, the node agent must be stopped. You can also delete a node agent if it has never been started, or never successfully able to contact the Domain Administration Server (that is, if it is still unbound).

Run the asadmin command delete-node-agent on the system where the node agent resides to delete the node agent files.

For a complete description of the command syntax, see the online help for the command.

For example:

asadmin delete-node-agent nodeagent1

where nodeagent1 is your node agent.

When deleting a node agent, you must also delete its configuration from the Domain Administration Server using either the Admin Console or the asadmin delete-node-agent-config command.

To View General Node Agent Information

1. In the tree component, select the Node Agents node.

2. Click the name of a node agent.

   If a node agent already exists but does not appear here, start the node agent on the node agent’s host machine using asadmin start-node-agent. See Starting a Node Agent

3. Check the node agent’s host name.

   If the host name is Unknown Host, then the node agent has not made initial contact with the Domain Administration Server (DAS).

4. Check the node agent status.

   The status can be:

   • Running: The node agent has been properly created and is currently running.

   • Not Running: Either the node agent has been created on the local machine, but never started or the node agent was started but has been stopped.

   • Waiting for Rendezvous: The node agent is a placeholder that has never been created on the local machine.

   See Creating a Node Agent and Starting a Node Agent.

5. Choose whether to start instances on start up.

   Select Yes to start server instances associated with the node agent automatically when the node agent is started. Select No to start the instances manually.

6. Determine whether the node agent has made contact with the Domain Administration Server.

   If the node agent has never made contact with the Domain Administration Server, it has never been successfully started.

7. Manage server instances associated with the node agent.

   If the node agent is running, start or stop an instance by clicking the checkbox next to the instance name and clicking Start or Stop.

To Delete a Node Agent Configuration

Through the Admin Console you can only delete the node agent configuration from the domain. You cannot delete the actual node agent. To delete the node agent itself, run the asadmin command delete-node-agent on the node agent’s local machine. For more information, see Deleting a Node Agent.

Before deleting the node agent configuration, the node agent must be stopped and it must not have any associated instances. To stop a node agent, use the asadmin command stop-node-agent. See Stopping a Node Agent for more information.

1. In the tree component, select the Node Agents node.

2. On the Node Agents page, select the checkbox next to the node agent to be deleted.

3. Click delete.

Equivalent asadmin command

delete-node-agent-config

To Edit a Node Agent Configuration

1. In the tree component, expand the Node Agents node.

2. Select the node agent configuration to edit.

3. Check Start Instances on Startup to start the agent’s server instances when the agent is started.

   You can also manually start and stop instances from this page.

   If this configuration is for a placeholder node agent, when you create the actual node agent using asadmin create-node-agent , it picks up this configuration. For information on creating a node agent, see Creating a Node Agent.

   If this configuration is for an existing node agent, the node agent configuration information is synchronized automatically.

To Edit a Node Agent Realm

You must set an authentication realm for users connecting to the node agent. Only administration users should have access to the node agent.

1. In the tree component, expand the Node Agents node.

2. Select the node agent configuration to edit.

3. Click the Auth Realm tab.

4. On the Node Agents Edit Realm page, enter a realm.

   The default is admin-realm, created when you create the node agent. To use a different realm, replace the realms in all the components controlled by the domain or the components won’t communicate properly.

5. In the Class Name field, specify the Java class that implements the realm.

6. Add any required properties.

   Authentication realms require provider-specific properties, which vary depending on what a particular implementation needs.

To Edit the Node Agent’s Listener for JMX

The node agent uses JMX to communicate with the Domain Administration Server. Therefore it must have a port to listen on for JMX requests, and other listener information.

1. In the tree component, expand the Node Agents node.

2. Select the node agent configuration to edit.

3. Click the JMX tab.

4. In the Address field, enter an IP address or host name.

   Enter 0.0.0.0 if the listener listens on all IP addresses for the server using a unique port value. Otherwise, enter a valid IP address for the server.

5. In the Port field, type the port on which the node agent’s JMX connector will listen.

   If the IP address is 0.0.0.0, the port number must be unique.

6. In the JMX Protocol field, type the protocol that the JMX connector supports.

   The default is rmi_jrmp.

7. Click the checkbox next to Accept All Addresses to allow a connection to all IP addresses.

   The node agent listens on a specific IP address associated to a network card or listens on all IP addresses. Accepting all addresses puts the value 0.0.0.0 in the “listening host address” property.

8. In the Realm Name field, type the name of the realm that handles authentication for the listener.

   In the Security section of this page, configure the listener to use SSL, TLS, or both SSL and TLS security.

   To set up a secure listener, do the following:

9. Check the Enabled box in the Security field.

   Security is enabled by default.

10. Set client authentication.

    To require clients to authenticate themselves to the server when using this listener, check the Enabled box in the Client Authentication field.

11. Enter a certificate nickname.

    Enter the name of an existing server keypair and certificate in the Certificate NickName field.

    For information about working with certificates and SSL, see the Admin Console online help.

12. In the SSL3/TLS section:

    1. Check the security protocol(s) to enable on the listener.

       You must check either SSL3 or TLS, or both protocols.

    2. Check the cipher suite used by the protocol(s).

       To enable all cipher suites, check All Supported Cipher Suites.

13. Click Save.

To Create a Standalone Server Instance

1. In the tree component, select the Stand—Alone Instances node.

2. Click New.

3. On the New Stand-Alone Server Instance page, provide the name of the instance.

4. Choose the node agent on which this instances is to be created and click OK.

Support for Installing DAS, Node Agent, and Server Instance in a DMZ

Before the release of Enterprise Server 2.1 patch 02, the DAS, Node Agent and Server Instance listens on dynamic port for JMX requests, posing a problem when these components are installed in the DMZ. The ports that need to open in the firewall would become dynamic and a protocol could not be set.

In Enterprise Server 2.1 patch 02 and all subsequent releases, this issue is resolved by the use of configurable parameters for each components. These parameters enable the components to listen for JMX requests at a configured port.

For DAS and server instance, define the following JVM option in the domain.xml file:

<jvm—options>Dcom.sun.aas.rmijmx.cbport=XXXX</jvm—options>

<jvm—options>-Dcom.sun.aas.jconsole.cbport=YYYY</jvm—options>

For node agents, the following properties need to be added to the nodeagent.properties file:

agent.rmijmx.cbport=XXXX

agent.jconsole.cbport=YYYY