Skip Navigation Links
Exit Print View
| Oracle Solaris Cluster Data Services Developer's Guide Oracle Solaris Cluster 4.1

Search Scope:

- This Document
- Entire Library

## Defining a Fault Monitor

The sample application implements a basic fault monitor to monitor the reliability of the DNS resource (`in.named`).

The fault monitor consists of the following elements:

- `dns_probe`, a user-defined program that uses `nslookup` to verify that the DNS resource that is controlled by the sample data service is running. If DNS is not running, this method attempts to restart it locally, or depending on the number of restart attempts, requests that the RGM relocate the data service to a different node.

- `dns_monitor_start`, a callback method that starts `dns_probe`. The RGM automatically calls `dns_monitor_start` after the sample data service is brought online if monitoring is enabled.

- `dns_monitor_stop`, a callback method that stops `dns_probe`. The RGM automatically calls `dns_monitor_stop` before bringing the sample data service offline.

- `dns_monitor_check`, a callback method that calls the `Validate` method to verify that the configuration directory is available when the `PROBE` program fails over the data service to a new node.

### How the Probe Program Works

The `dns_probe` program implements a continuously running process that verifies that the DNS resource that is controlled by the sample data service is running. The `dns_probe` is started by the `dns_monitor_start` method, which is automatically run by the RGM after the sample data service is brought online. The data service is stopped by the `dns_monitor_stop` method, which the RGM runs before the RGM brings the sample data service offline.

This section describes the major pieces of the `PROBE` method for the sample application. It does not describe functionality that is common to all callback methods, such as the `parse_args()` function. This section also does not describe using the `syslog()` function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the `PROBE` method, see `PROBE` Program Code Listing.

#### What the Probe Program Does

The probe runs in an infinite loop. It uses `nslookup` to verify that the correct DNS resource is running. If DNS is running, the probe sleeps for a prescribed interval (set by the `Thorough_probe_interval` system-defined property) and checks again. If DNS is not running, this program attempts to restart it locally, or depending on the number of restart attempts, requests that the RGM relocate the data service to a different node.

#### Obtaining Property Values

This program requires the values of the following properties:

- `Thorough_probe_interval` – To set the period during which the probe sleeps

- `Probe_timeout` – To enforce the timeout value of the probe on the `nslookup` command that does the probing

- `Network_resources_used` – To obtain the IP address on which DNS is running

- `Retry_count` and `Retry_interval` – To determine the number of restart attempts and the period over which to count them

- `RT_basedir` – To obtain the directory that contains the `PROBE` program and the `gettime` utility

The `scha_resource_get()` function obtains the values of these properties and stores them in shell variables, as follows:

    PROBE_INTERVAL=`scha_resource_get -O Thorough_probe_interval \
    -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME`

    PROBE_TIMEOUT_INFO=`scha_resource_get -O Extension -R $RESOURCE_NAME \
    -G $RESOURCEGROUP_NAME Probe_timeout`
    Probe_timeout=`echo $probe_timeout_info | awk '{print $2}'`

    DNS_HOST=`scha_resource_get -O Network_resources_used -R $RESOURCE_NAME \
    -G $RESOURCEGROUP_NAME`

    RETRY_COUNT=`scha_resource_get -O Retry_count -R $RESOURCE_NAME -G \
    $RESOURCEGROUP_NAME`

    RETRY_INTRVAL=`scha_resource_get -O Retry_interval -R $RESOURCE_NAME -G \
    $RESOURCEGROUP_NAME`

    RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
     $RESOURCEGROUP_NAME`

Note - For system-defined properties, such as `Thorough_probe_interval`, the `scha_resource_get()` function returns the value only. For extension properties, such as `Probe_timeout`, the `scha_resource_get()` function returns the type and value. Use the `awk` command to obtain the value only.

#### Checking the Reliability of the Service

The probe itself is an infinite `while` loop of `nslookup` commands. Before the `while` loop, a temporary file is set up to hold the `nslookup` replies. The `probefail` and `retries` variables are initialized to 0.

    # Set up a temporary file for the nslookup replies.
    DNSPROBEFILE=/var/cluster/run/.$RESOURCE_NAME.probe
    probefail=0
    retries=0

The `while` loop carries out the following tasks:

- Sets the sleep interval for the probe

- Uses `/usr/cluster/bin/hatimerun` to start `nslookup`, passes the `Probe_timeout` value, and identifies the target host

- Sets the `probefail` variable based on the success or failure of the `nslookup` return code

- If `probefail` is set to 1 (failure), verifies that the reply to `nslookup` came from the sample data service and not some other DNS server

Here is the `while` loop code.

    while :
    do
       # The interval at which the probe needs to run is specified in the
       # property THOROUGH_PROBE_INTERVAL. Therefore, set the probe to sleep
       # for a duration of THOROUGH_PROBE_INTERVAL.
       sleep $PROBE_INTERVAL

       # Run an nslookup command of the IP address on which DNS is serving.
       /usr/cluster/bin/hatimerun -t $PROBE_TIMEOUT /usr/sbin/nslookup $DNS_HOST $DNS_HOST \
       > $DNSPROBEFILE 2>&1

          retcode=$?
          if [ $retcode -ne 0 ]; then
                probefail=1
          fi

       # Make sure that the reply to nslookup comes from the HA-DNS
       # server and not from another nameserver mentioned in the
       # /etc/resolv.conf file.
       if [ $probefail -eq 0 ]; then
    # Get the name of the server that replied to the nslookup query.
       SERVER=` awk ' $1=="Server:" { print $2 }' \
       $DNSPROBEFILE | awk -F. ' { print $1 } ' `
       if [ -z "$SERVER" ]; then
          probefail=1
          else
             if [ $SERVER != $DNS_HOST ]; then
                probefail=1
             fi
       fi
    fi

#### Comparing Restart With Failover

If the `probefail` variable is something other than 0 (success), the `nslookup` command timed out or the reply came from a server other than the sample service's DNS. In either case, the DNS server is not functioning as expected and the fault monitor calls the `decide_restart_or_failover()` function to determine whether to restart the data service locally or request that the RGM relocate the data service to a different node. If the `probefail` variable is 0, a message is generated that the probe was successful.

       if [ $probefail -ne 0 ]; then
             decide_restart_or_failover
       else
             logger -p ${SYSLOG_FACILITY}.err\
             -t [$SYSLOG_TAG]\
             "${ARGV0} Probe for resource HA-DNS successful"
       fi

The `decide_restart_or_failover()` function uses a time window (`Retry_interval`) and a failure count (`Retry_count`) to determine whether to restart DNS locally or request that the RGM relocate the data service to a different node. This function implements the following conditional logic. The code listing for `decide_restart_or_failover()` in `PROBE` Program Code Listing contains the code.

- If this is the first failure, restart the data service. Log an error message and bump the counter in the `retries` variable.

- If this is not the first failure, but the window has been exceeded, restart the data service. Log an error message, reset the counter, and slide the window.

- If the time is still within the window and the retry counter has been exceeded, fail over to another node. If the failover does not succeed, log an error and exit the probe program with status 1 (failure).

- If time is still within the window but the retry counter has not been exceeded, restart the data service. Log an error message and bump the counter in the `retries` variable.

If the number of restarts reaches the limit during the time interval, the function requests that the RGM relocate the data service to a different node. If the number of restarts is under the limit, or the interval has been exceeded so the count begins again, the function attempts to restart DNS on the same node.

Note the following points about this function:

- The `gettime` utility is used to track the time between restarts. This is a C program that is located in the (`RT_basedir`) directory.

- The `Retry_count` and `Retry_interval` system-defined resource properties determine the number of restart attempts and the time interval over which to count. These properties default to two attempts in a period of 5 minutes (300 seconds) in the RTR file, although the cluster administrator can change these values.

- The `restart_service()` function is called to attempt to restart the data service on the same node. See the next section, Restarting the Data Service, for information about this function.

- The `scha_control()` API function, with the `SCHA_GIVEOVER` argument, brings the resource group that contains the sample data service offline and back online on a different node.

#### Restarting the Data Service

The `restart_service()` function is called by `decide_restart_or_failover()` to attempt to restart the data service on the same node.

This function executes the following logic:

- Determines if the data service is still registered under the PMF.

  If the service is still registered, the function carries out the following actions:

  - Obtains the `Stop` method name and the `Stop_timeout` value for the data service

  - Uses `/usr/cluster/bin/hatimerun` to start the `Stop` method for the data service, passing the `Stop_timeout` value

  - If the data service is successfully stopped, obtains the `Start` method name and the `Start_timeout` value for the data service

  - Uses `hatimerun` to start the `Start` method for the data service, passing the `Start_timeout` value

- If the data service is no longer registered under the PMF, the implication is that the data service has exceeded the maximum number of allowable retries under the PMF. The `scha_control` command is run with the `GIVEOVER` argument to fail over the data service to a different node.

    function restart_service
    {

            # To restart the data service, first verify that the
            # data service itself is still registered under PMF.
            pmfadm -q $PMF_TAG
            if [[ $? -eq 0 ]]; then
                    # Since the TAG for the data service is still registered under
                    # PMF, first stop the data service and start it back up again.

                    # Obtain the Stop method name and the STOP_TIMEOUT value for
                    # this resource.
                    STOP_TIMEOUT=`scha_resource_get -O STOP_TIMEOUT \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                    STOP_METHOD=`scha_resource_get -O STOP \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                    /usr/cluster/bin/hatimerun -t $STOP_TIMEOUT $RT_BASEDIR/$STOP_METHOD \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                            -T $RESOURCETYPE_NAME

                    if [[ $? -ne 0 ]]; then
                            logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                    “${ARGV0} Stop method failed.”
                            return 1
                    fi

                    # Obtain the START method name and the START_TIMEOUT value for
                    # this resource.
                    START_TIMEOUT=`scha_resource_get -O START_TIMEOUT \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                    START_METHOD=`scha_resource_get -O START \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                    /usr/cluster/bin/hatimerun -t $START_TIMEOUT $RT_BASEDIR/$START_METHOD \
                            -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                            -T $RESOURCETYPE_NAME

                    if [[ $? -ne 0 ]]; then
                            logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                    “${ARGV0} Start method failed.”
                            return 1
                    fi

            else
                    # The absence of the TAG for the dataservice
                    # implies that the data service has already
                    # exceeded the maximum retries allowed under PMF.
                    # Therefore, do not attempt to restart the
                    # data service again, but try to failover
                    # to another node in the cluster.
                    scha_control -O GIVEOVER -G $RESOURCEGROUP_NAME \
                            -R $RESOURCE_NAME
            fi

            return 0
    }

#### Probe Exit Status

The sample data service's `PROBE` program exits with failure if attempts to restart locally fail and the attempt to fail over to a different node fails as well. This program logs the message `Failover attempt failed`.

### How the `Monitor_start` Method Works

The RGM calls the `Monitor_start` method to start the `dns_probe` method after the sample data service is brought online.

This section describes the major pieces of the `Monitor_start` method for the sample application. This section does not describe functionality that is common to all callback methods, such as the `parse_args()` function. This section also does not describe using the `syslog()` function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the `Monitor_start` method, see `Monitor_start` Method Code Listing.

#### What the `Monitor_start` Method Does

This method uses the PMF (`pmfadm`) to start the probe.

#### Starting the Probe

The `Monitor_start` method obtains the value of the `RT_basedir` property to construct the full path name for the `PROBE` program. This method starts the probe by using the infinite retries option of `pmfadm` (`-n -1`, `-t -1`), which means that if the probe fails to start, the PMF tries to start it an infinite number of times over an infinite period of time.

    # Find where the probe program resides by obtaining the value of the
    # RT_basedir property of the resource.
    RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
    $RESOURCEGROUP_NAME`

    # Start the probe for the data service under PMF. Use the infinite retries
    # option to start the probe. Pass the resource name, type, and group to the
    # probe program.
    pmfadm -c $RESOURCE_NAME.monitor -n -1 -t -1 \
       $RT_BASEDIR/dns_probe -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
       -T $RESOURCETYPE_NAME

### How the `Monitor_stop` Method Works

The RGM calls the `Monitor_stop` method to stop execution of `dns_probe` when the sample data service is brought offline.

This section describes the major pieces of the `Monitor_stop` method for the sample application. This section does not describe functionality that is common to all callback methods, such as the `parse_args()` function. This section also does not describe using the `syslog()` function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the `Monitor_stop` method, see `Monitor_stop` Method Code Listing.

#### What the `Monitor_stop` Method Does

This method uses the PMF (`pmfadm`) to check whether the probe is running, and if so, to stop it.

#### Stopping the Monitor

The `Monitor_stop` method uses `pmfadm -q` to see if the probe is running, and if so, uses `pmfadm -s` to stop it. If the probe is already stopped, the method exits successfully anyway, which guarantees the idempotence of the method.

| Caution - Be certain to use the `KILL` signal with `pmfadm` to stop the probe and not a signal that can be masked, such as `TERM`. Otherwise, the `Monitor_stop` method can hang indefinitely and eventually time out. The reason is that the `PROBE` method calls `scha_control()` when it is necessary to restart or fail over the data service. When `scha_control()` calls `Monitor_stop` as part of the process of bringing the data service offline, if `Monitor_stop` uses a signal that can be masked, `Monitor_stop` hangs waiting for `scha_control()` to complete, and `scha_control()` hangs waiting for `Monitor_stop` to complete.

    # See if the monitor is running, and if so, kill it.
    if pmfadm -q $PMF_TAG; then
       pmfadm -s $PMF_TAG KILL
       if [ $? -ne 0 ]; then
             logger -p ${SYSLOG_FACILITY}.err \
                -t [$SYSLOG_TAG] \
                "${ARGV0} Could not stop monitor for resource " \
                $RESOURCE_NAME
               exit 1
       else
             # could successfully stop the monitor. Log a message.
             logger -p ${SYSLOG_FACILITY}.err \
                -t [$SYSLOG_TAG] \
                "${ARGV0} Monitor for resource " $RESOURCE_NAME \
                " successfully stopped"
       fi
    fi
    exit 0

#### `Monitor_stop` Exit Status

The `Monitor_stop` method logs an error message if it cannot stop the `PROBE` method. The RGM puts the sample data service into `MONITOR_FAILED` state on the primary node, which can panic the node.

`Monitor_stop` should not exit before the probe has been stopped.

### How the `Monitor_check` Method Works

The RGM calls the `Monitor_check` method whenever the `PROBE` method attempts to fail over the resource group that contains the data service to a new node.

This section describes the major pieces of the `Monitor_check` method for the sample application. This section does not describe functionality that is common to all callback methods, such as the `parse_args()` function. This section also does not describe using the `syslog()` function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the `Monitor_check` method, see `Monitor_check` Method Code Listing.

The `Monitor_check` method must be implemented so that it does not conflict with other methods that are running concurrently.

The `Monitor_check` method calls the `Validate` method to verify that the DNS configuration directory is available on the new node. The `Confdir` extension property points to the DNS configuration directory. Therefore, `Monitor_check` obtains the path and name for the `Validate` method and the value of `Confdir`. It passes this value to `Validate`, as shown in the following listing.

    # Obtain the full path for the Validate method from
    # the RT_basedir property of the resource type.
    RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME \
       -G $RESOURCEGROUP_NAMÈ

    # Obtain the name of the Validate method for this resource.
    VALIDATE_METHOD=`scha_resource_get -O Validate \
       -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ

    # Obtain the value of the Confdir property in order to start the
    # data service. Use the resource name and the resource group entered to
    # obtain the Confdir value set at the time of adding the resource.
    config_info=`scha_resource_get -O Extension -R $RESOURCE_NAME \
     -G $RESOURCEGROUP_NAME Confdir`

    # scha_resource_get returns the type as well as the value for extension
    # properties. Use awk to get only the value of the extension property.
    CONFIG_DIR=`echo $config_info | awk `{print $2}'`

    # Call the validate method so that the dataservice can be failed over
    # successfully to the new node.
    $RT_BASEDIR/$VALIDATE_METHOD -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
       -T $RESOURCETYPE_NAME -x Confdir=$CONFIG_DIR

See How the `Validate` Method Works to see how the sample application verifies the suitability of a node for hosting the data service.

Copyright © 2000, 2012, Oracle and/or its affiliates. All rights reserved. Legal Notices |