Sun Java System Application Server Enterprise Edition 8.1 2005Q2 High Availability Administration Guide

- Previous: Chapter 2 Installing and Setting Up High Availability Database
- Next: Chapter 4 Configuring Load Balancing and Failover

# Chapter 3 Administering High Availability Database

This chapter describes the high availability database (HADB) in the Sun Java System Application Server Enterprise Edition environment. It explains how to configure and administer the HADB. Before you can create and administer the HADB, you must first determine the topology of your systems and install the HADB software on the various machines.

This chapter discusses the following topics:

- Using the HADB Management Agent

- Using the hadbm Management Command

- Configuring HADB

- Managing HADB

- Expanding HADB

- Monitoring HADB

- Maintaining HADB Machines

# Using the HADB Management Agent

The management agent, `ma`, executes management commands on HADB hosts. The management agent also ensures availability of the HADB node supervisor processes by restarting them if they fail.

- Management Agent Command Syntax

- Customizing Management Agent Configuration

- Starting the Management Agent

## Management Agent Command Syntax

The syntax of the management agent `ma` command is:

    ma [common-options]
    [ service-options]
    config-file

Where:

- common-options is one or more of the common options described in Management Agent Command Syntax.

- service-options is one of the Windows service options described in Management Agent Command Syntax.

- config-file is the full path to the management agent configuration file. For more information, see Customizing Management Agent Configuration.

Table 3–1 Management Agent Common Options

Option | Description | Default
--- | --- | ---
--define name=value-D | Assign value to property name, where property is one of the properties defined in Configuration File. This option can be repeated multiple times. | None
--help-? | Display help information. | False
--javahome path-j | Use Java Runtime Environment (1.4 or later) located at path. | None
--systemroot path-y | Path to the operating system root as normally set in %SystemRoot%. | None
--version-V | Display version information. | False

Management Agent Command Syntax describes options for starting the management agent as a Windows service. The -i, -r, and -s options are mutually exclusive; that is, use only one of them at a time.

On Windows, when specifying paths for property values in the configuration file or on the command line, escape file paths containing spaces with double quotes (`"`). Escape colon (`:`) drive separators, , and backslash (`\`) directory separators, with double quotes and a backslash, like this: `"\:` and `"\\`.

Table 3–2 Management Agent Service Options (Windows Only)

Option | Description | Default
--- | --- | ---
--install-i | Install the agent as a Windows service and start the service. Use only one of -i, -r, and -s options. | False
--name servicename-n | Use specified name for the service when running multiple agents on a host. | HADBMgmtAgent
--remove-r | Stop the service and delete the agent from the Windows service manger. Use only one of -i, -r, and -s options. | False
--service-s | Run the agent as a Windows service. Use only one of -i, -r, and -s options. | False

## Customizing Management Agent Configuration

HADB includes a configuration file that you can use to customize the management agent settings. When you start the management agent without specifying a configuration file, it uses default values. If you specify a configuration file, the management agent will use the settings in that file. You can re-use the configuration file on all hosts in a domain.

### To customize the management agent configuration on each HADB host

1. Edit the management agent configuration file and set the values as desired.

2. Start the management agent, specifying the customized configuration file as the argument.

### Configuration File

With Java Enterprise System, all the entries in the configuration file are commented out. No changes are required to use the default configuration. To customize the management agent configuration, remove the comments from the file, change the values as desired, then start the management agent specifying the configuration file as an argument.

The management agent configuration file is installed to:

- Solaris and Linux: `/etc/opt/SUNWhadb/mgt.cfg`.

- Windows: install_dir `\lib\mgt.cfg`.

With the standalone installer, the management agent configuration file is installed to:

- Solaris and Linux: HADB_install_dir `/bin/ma.cfg` .

- Windows: HADB_install_dir `\bin\ma.cfg`.

The following table describes the settings in the configuration file.

Table 3–3 Configuration File Settings

Setting Name | Description | Default
--- | --- | ---
    console.loglevel
| Console log level. Valid values are SEVERE, ERROR, WARNING, INFO, FINE, FINER, FINEST | WARNING
    logfile.loglevel
| Log file log level. Valid values are SEVERE, ERROR, WARNING, INFO, FINE, FINER, FINEST | INFO
    logfile.name
| Name and location of log file. Must be a valid path with read/write access. | Solaris and Linux:`/var/opt/SUNWhadb/ma/ma.log` Windows: `HADB_install_dir\ma.log`
    ma.server.type
| Client protocol. Only JMXMP is supported. | jmxmp
    ma.server.
    jmxmp.port
| Port number for internal (UDP) and external (TCP) communication. Must be a positive integer. Recommended range is 1024-49151. | 1862
    ma.server.
    mainternal.interfaces
| Interfaces for internal communication for machines with multiple interfaces. Must be a valid IPv4 address mask. All management agents in a domain must use the same subnet For example, if a host has two interfaces, 10.10.116.61 and 10.10.124.61, use 10.10.116.0/24 to use the first interface. The number after the slash indicates the number of bits in the subnet mask. | None
    ma.server.
    dbdevicepath
| Path to store HADB device information. | Solaris and Linux: `/var/opt/SUNWhadb/4` Windows: HADB_install_dir `\device`
    ma.server.
    dbhistorypath
| Path to store HADB history files. | Solaris and Linux: `/var/opt/SUNWhadb` Windows: REPLACEDIR (replaced by the actual URL at runtime.)
    ma.server.
    dbconfigpath
| Path to store node configuration data. | Solaris and Linux: `/var/opt/SUNWhadb/dbdef` Windows: `C:\Sun\SUNWhadb\dbdef`
    repository.dr.path
| Path to domain repository files. | Solaris and Linux: `/var/opt/SUNWhadb/repository` Windows: `C:\Sun\SUNWhadb\repository`

## Starting the Management Agent

You can start the management agent two ways:

- As a service, for production use. See Starting the Management Agent as a Service To ensure availability of the management agent, make sure it is restarted automatically when the system reboots. See Ensuring Automatic Restart of the Management Agent.

- As a regular process (in console mode), for evaluation, testing, or development. See Starting the Management Agent in Console Mode.

  In each case, the procedures are different depending on whether you are using Java Enterprise System or the standalone Application Server.

### Starting the Management Agent as a Service

Starting the management agent as a service ensures that it will continue to run until the system shuts down or you explicitly stop it.

#### Starting the Management Agent as a Service with Java Enterprise System on Solaris or Linux

To start the management agent as a service, use this command:

    /etc/init.d/ma-initd start

To stop the service, use this command:

    /etc/init.d/ma-initd stop

#### Starting the Management Agent as a Service with Java Enterprise System on Windows

To start the management agent as a Windows service, use this command: `HADB_install_dir\bin\ma -i [config-file ]`

The optional argument config-file specifies the management agent configuration file. Use a configuration file only if you want to change the default management agent configuration.

To stop the management agent and remove (deregister) it as a service, use the command: `HADB_install_dir\bin\ma -r [config-file ]`

To perform administration, choose Administrative Tools | Services, which enables you to start and stop the service, disable automatic startup, and so on.

#### Starting the Management Agent as a Service with Standalone Application Server on Solaris or Linux

To start the management agent as a service, use this command:

    HADB_install_dir/bin/ma-initd start

To stop the service, use this command:

    HADB_install_dir/bin/ma-initd stop

#### Starting the Management Agent as a Service with Standalone Application Server on Windows

To start the management agent as a Windows service, use this command: `HADB_install_dir\bin\ma -i [config-file ]`

The optional argument config-file specifies the management agent configuration file. Use a configuration file only if you want to change the default management agent configuration.

To stop the management agent and remove (deregister) it as a service, use the command: `HADB_install_dir\bin\ma -r [config-file ]`

To perform administration, choose Administrative Tools | Services, which enables you to start and stop the service, disable automatic startup, and so on.

### Ensuring Automatic Restart of the Management Agent

On Windows platforms, once you have started the management agent as a service, use the Windows administrative tools to set the service Startup type to “Automatic,” and the desired Recovery options.

On Solaris and Linux platforms, use the procedures in this section to ensure the availability of the management agent in case the `ma` process fails or the operating system reboots. Doing so is appropriate for a production deployment.

The following procedures ensure the management agent starts only when the system enters:

- Runlevel 3 on Solaris (the default).

- Runlevel 5 on RedHat Linux (the default in graphical mode).

Entering other runlevels stops the management agent.

#### To Configure automatic restart with Java Enterprise System on Solaris or Linux

##### Before You Begin

This section assumes you have a basic understanding of operating system initialization and runlevels. For information on these topics, see your operating system documentation.

1. Ensure that your system has a default runlevel of 3 or 5.

   To check the default runlevel of your system, inspect the file `/etc/inittab` , and look for a line near the top similar to this:

       id:5:initdefault:

   This example shows a default runlevel of 5.

2. Create the following softlinks to the file `/etc/init.d/ma-initd` :

       /etc/rc0.d/K20ma-initd
       /etc/rc1.d/K20ma-initd
       /etc/rc2.d/K20ma-initd
       /etc/rc3.d/S99ma-initd
       /etc/rc5.d/S99ma-initd
       /etc/rcS.d/K20ma-initd

3. Reboot the machine.

##### Next Steps

To deactivate automatic start and stop of the agent, remove the links or change the letters K and S in the link names to lowercase.

#### To Configure automatic restart with Standalone Application Server on Solaris or Linux

1. In a shell, change your current directory to HADB_install_dir `/bin`.

2. Edit the shell script `ma-initd`.

   Make sure the default values of HADB_ROOT and HADB_MA_CFG in the script to reflect your installation:

   - HADB_ROOT is the HADB installation directory, HADB_install_dir.

   - HADB_MA_CFG is the location of the management agent configuration file. For more information, see Customizing Management Agent Configuration

3. Copy `ma-initd` to the directory `/etc/init.d`

4. Create the following soft links to the file `/etc/init.d/ma-initd` :

       /etc/rc0.d/K20ma-initd
       /etc/rc1.d/K20ma-initd
       /etc/rc2.d/K20ma-initd
       /etc/rc3.d/S99ma-initd
       /etc/rc5.d/S99ma-initd
       /etc/rcS.d/K20ma-initd

##### Next Steps

To deactivate automatic start and stop of the agent, remove the links or change the letters K and S in the link names to lowercase.

### Starting the Management Agent in Console Mode

You may wish to start the management agent manually in console mode for evaluation or testing. Do not start the management agent this way in a production environment, because the `ma` process will not restart after a system or process failure and will terminate when the command window is closed.

#### Starting management agent in Console Mode with Java Enterprise System on Solaris or Linux

To start the HADB management agent in console mode, use the command:

    opt/SUNWhadb/bin/ma [config-file]

The default management agent configuration file is `/etc/opt/SUNWhadb/mgt.cfg`

To stop the management agent, kill the process or close the shell window.

#### Starting management agent in Console Mode with Java Enterprise System on Windows

To start the management agent in console mode, use the command:

    HADB_install_dir\bin\ma [config-file]

The optional argument config-file is the name of the management agent configuration file. For more information on the configuration file, see Customizing Management Agent Configuration.

To stop the agent, kill the process.

#### Starting management agent in Console mode with Standalone Application Server on Windows

To start the management agent in console mode, use the command:

    HADB_install_dir\bin\ma [config-file]

The optional argument config-file is the name of the management agent configuration file; for more information, see Customizing Management Agent Configuration

To stop the management agent, kill the process.

#### Starting management agent in Console Mode with Standalone Application Server on Solaris or Linux

To start the HADB management agent in console mode, use the command:

    HADB_install_dir/bin/ma [config-file]

The default management agent configuration file is HADB_install_dir `/bin/ma.cfg`

To stop the management agent, kill the process or close the shell window.

# Using the hadbm Management Command

Use the `hadbm` command-line utility to manage an HADB domain, its database instances, and nodes. The `hadbm` utility (also called the management client) sends management requests to the specified management agent, acting as a management server, which has access to the database configuration from the repository.

This section describes the hadbm command-line utility, with the following topics:

- Command Syntax

- Security Options

- General Options

- Environment Variables

## Command Syntax

The `hadbm` utility is located in the HADB_install_dir `/bin` directory. The general syntax of the `hadbm` command is:

    hadbm subcommand
    [-short-option [option-value]]
    [--long-option [option-value]]
    [operands]

The subcommand identifies the operation or task to perform. Subcommands are case-sensitive. Most commands have one operand (usually `dbname),` but some have none, and some have two.

Options modify how `hadbm` performs a subcommand. Options are case-sensitive. Each option has a long form and a short form. Precede the short form with a single dash (`-`); precede the long forms with two dashes (`--`). Most options require argument values, except for boolean options, which must be present to switch a feature on. Options are not required for successful execution of the command.

If a subcommand requires a database name, and you do not specify one, `hadbm` will use the default database, `hadb`.

##### Example 3–1 Example of hadbm command

The following illustrates the `status` subcommand:

    hadbm status --nodes

## Security Options

For security reasons, all `hadbm` commands require an administrator password. Use the `--adminpassword` option to set the password when you create a database or domain. From then on, you must specify that password when you perform operations on the database or domain.

For enhanced security, use the `--adminpasswordfile` option to specify a file containing the password, instead of entering it on the command line. Define the password in the password file with the following line:

`HADBM_ADMINPASSWORD=`password

Replace password with the password. Any other content in the file is ignored.

If you specify both the `--adminpassword` and `--adminpasswordfile` options, the `--adminpassword` takes precedence. If a password is required, but is not specified in the command, `hadbm` prompts you for a password.

Note –

You can change the administrator password, if required, using the command `setadminpassword`. For more information about the command, see the man page for the command `setadminpassword`.

In addition to the administrator password, HADB also requires a database password to perform operations that modify the database schema. You must use both passwords when using the following commands: `hadbm create`, `hadbm addnodes`, and `hadbm refragment`.

Specify the database password on the command line with the `--dbpassword` option. Similar to the administrator password, you can also put the password in a file and use the `--dbpasswordfile` option, specifying the file location. Set the password in the password file with the following line:

    HADBM_DBPASSWORD=password

For testing or evaluation, you can turn off password authentication with the `--no-adminauthentication` option when you create a database or domain. For more information, see Creating a Database and Creating a Management Domain

The following table summarizes the `hadbm` security command line options.

Table 3–4 `hadbm` Security Options

Option (Short Form) | Description
--- | ---
--adminpassword=password -w | Specifies administrator password for the database or domain. If you use this option when you create a database or domain, then you must provide the password each time you use hadbm to operate on the database or domain. Use either this option or --adminpasswordfile, but not both.
--adminpasswordfile=filepath -W | Specifies file that contains the administrator password for the database or domain. If you use this option when you create a database or domain, then you must provide the password each time you use hadbm to operate on the database or domain. Use either this option or --adminpassword, but not both.
--no-adminauthentication -U | Use this option when you create a database or domain to specify that no administrator password is required. For security reasons, do not use this option in a production deployment.
--dbpassword= password -p | Specifies the database password. If you use this option when you create the database, then you must provide the password each time you use an `hadbm` command to operate on the database. Creates a password for the HADB system user. Must be at least 8 characters. Use either this option or --dbpasswordfile, but not both.
--dbpasswordfile= filepath -P | Specifies a file that contains the password for the HADB system user. Use either this option or --dbpassword, but not both.

## General Options

General command options can be used with any `hadbm` subcommand. All are boolean options that are false by default. The following table describes the `hadbm` general command options.

Table 3–5 `hadbm` General Options

Option(Short Form) | Description
--- | ---
--quiet -q | Execute the subcommand silently without any descriptive messages.
--help -? | Display a brief description of this command and all the supported subcommands. No subcommand is required.
--version -V | Display the version details of the `hadbm` command. No subcommand is required.
--yes -y | Execute the subcommand in non-interactive mode.
--force -f | Execute the command non-interactively and does not throw an error if the command’s post condition is already achieved.
--echo -e | Display the subcommand with all the options and their user-defined values or the default values, then executes the subcommand.
--agent=URL -m | URL to the management agents. URL is: hostlist:port, where hostlist is a comma separated list of hostnames or IP-addresses, and port is the port number on which the management agent is operating. Default is localhost:1862. NOTE: This option is not valid with `hadbm addnodes`.

## Environment Variables

For convenience, you can set an environment variable instead of specifying a command option. The following table describes environment variables that correspond to `hadbm` command options.

Table 3–6 HADB Options and Environment Variables

Long Form | Short Form | Default | Environment Variable
--- | --- | --- | ---
--adminpassword | -w | none | $HADBM_ADMINPASSWORD
--agent | --m | localhost:1862 | $HADBM_AGENT
--datadevices | -a | `1` | $HADBM_DATADEVICES
dbname | none | `hadb` | $HADBM_DB
--dbpassword | -p | none | $HADBM_DBPASSWORD
--dbpasswordfile | -P | none | $HADBM_DBPASSWORDFILE
--devicepath | -d | Solaris and Linux: `/var/opt/SUNWhadb` Windows: `C:\Sun\AppServer\SUNWhadb\vers`, where vers is the HADB version number. | $HADBM_DEVICEPATH
--devicesize | -z | none | $HADBM_DEVICESIZE
--echo | -e | False | $HADBM_ECHO
--fast | -F | False | $HADBM_FAST
--force | -f | False | $HADBM_FORCE
--help | -? | False | $HADBM_HELP
--historypath | -t | Solaris and Linux: `/var/opt/SUNWhadb` Windows: REPLACEDIR, replaced by the actual URL at runtime. | $HADBM_HISTORYPATH
--hosts | -H | none | $HADBM_HOSTS
--interactive | -i | True | $HADBM_INTERACTIVE
--no-refragment | -r | False | $HADBM_NOREFRAGMENT
--portbase | -b | 15200 | $HADBM_PORTBASE
--quiet | -q | False | $HADBM_QUIET
--repair | -R | True | $HADBM_REPAIR
--rolling | -g | True | $HADBM_ROLLING
--saveto | -o | none | $HADBM_SAVETO
--set | -S | none | $HADBM_SET
--spares | -s | 0 | $HADBM_SPARES
--startlevel | -l | normal | $HADBM_STARTLEVEL
--version | -V | False | $HADBM_VERSION
--yes | -y | False | $HADBM_YES

# Configuring HADB

This section describes the following basic HADB configuration tasks:

- Creating a Management Domain

- Creating a Database

- Viewing and Modifying Configuration Attributes

- Configuring the JDBC Connection Pool

## Creating a Management Domain

The command `hadbm createdomain` creates a management domain containing the specified HADB hosts. The command initializes internal communication channels between hosts and the persistence configuration store.

The syntax of the command is:

    hadbm createdomain
     [--adminpassword=password |--adminpasswordfile=
    file | --no-adminauthentication] [--agent=maurl]
     hostlist

The hostlist operand is a comma-separated list of HADB hosts, each of which is a valid IPv4 network address. Include all the hosts that you want to be in the new domain in the hostlist.

See General Options for a description of the command options.

Before using this command, be sure an HADB management agent is running on every host in the hostlist. Additionally, the management agents must:

- Not be members of an existing domain.

- Be configured to use the same port.

- Be able to reach each other over UDP, TCP, and with IP multicast.

After `hadbm` creates the management domain, it enables all the hosts in the domain. Then the management agents are ready to manage databases. After creating HADB domains, the next step is to create the HADB database. For more information on creating HADB databases, see Creating a Database .

##### Example 3–2 Creating an HADB Management Domain

The following example creates a management domain on the four specified hosts:

`hadbm createdomain` `--adminpassword=` password `host1,host2,host3,host4`

After `hadbm` successfully executes the command, you will see the message:

    Domain host1,host2,host3, host4 created.

After creating HADB domains, register the path and version of the HADB packages with the management agents.

## Creating a Database

Use the `hadbm create` command to create a database manually.

Before you use this command to create a database, create the management domain and register the HADB package. If you have not performed these two steps when you run `hadbm create` , it implicitly performs them. Although this might seem like less work, failures in any of the commands can make debugging difficult. Besides, `hadbm create` is not atomic, that is, if one of the implicit commands fails, the commands that executed successfully will not be rolled back. Therefore, it is best to create the database only after creating the domain and registering the HADB package.

For example, if `hadbm createdomain` and `hadbm registerpackage` execute successfully but `hadbm create database` fails, the changes made by `hadbm createdomain` and `hadbm registerpackage` will persist.

### To create a database

1. Create the management domain.

   For more information, see Creating a Management Domain

2. Register the HADB package.

   For more information, see Registering HADB Packages for more information.

3. Use the hadbm create command to create the database.

   For information on command syntax, see the following section.

### `hadbm create` Command Syntax

`hadbm create [`--package`=name] [`--packagepath`=path] [`--historypath`=path] [`--devicepath`=path] [`--datadevices`=number ] [`--portbase`=number] [`--spares`=number] [`--set`=attr-val-list] [`--agent`=maurl] [`--no-cleanup`] [`--no-clear` ] [`--devicesize` =size] [`--dbpassword`=password | `--dbpasswordfile`=file ] `--hosts`=host list [`--adminpassword`=password | `--adminpasswordfile`=file | `--no-adminauthentication` ]` [dbname]

The dbname operand specifies the database name, which must be unique. To make sure the database name is unique, use the `hadbm list` command to list existing database names. Use the default database name unless you need to create multiple databases. For example, to create multiple clusters with independent databases on the same set of HADB machines, use a separate database name for each cluster.

The `hadbm create` command writes error messages to the console, not log files.

Table 3–7 describes the special `hadbm create` command options. See General Options for a description of additional command options.

Table 3–7 hadbm create Options

Option(Short Form) | Description | Default
--- | --- | ---
--datadevices= number -a | Number of data devices on each node, between one and eight inclusive. Data devices are numbered starting at 0. | 1
--devicepath= path -d | Path to the devices. There are four devices: `DataDevice` `NiLogDevice` (node internal log device) `RelalgDevice` (relational algebra query device) `NoManDevice` (node manager device). This path must exist and be writable. To set this path differently for each node or each device, see Setting Heterogeneous Device Paths | Solaris and Linux: `/var/opt/SUNWhadb` Windows: `C:\Sun\AppServer\SUNWhadb\vers`, where vers is the HADB version number. Default is specified by `ma.server.dbdevicepath in` management agent configuration file. For more details, see Configuration File
--devicesize= size -z | Device size for each node. For more information, see Specifying Device Size. Increase the device size as described in Adding Storage Space to Existing Nodes | 1024MB Maximum size is lesser of maximum operating system file size or 256 GB. Minimum size is: (4 x `LogbufferSize` + 16MB) / n Where n is the number of data devices given by the option `--datadevices`.
--historypath= path -t | Path to the history files. This path must already exist and be writable. For more information on history files, see Clearing and Archiving History Files | Default is specified by `ma.server.dbhistorypath` in management agent configuration file. For details, see Configuration File Solaris and Linux:`/var/opt/SUNWhadb` On Windows: REPLACEDIR (replaced by the actual URL at runtime.)
--hosts= hostlist -H | Comma-separated list of host names or IP addresses (IPv4 only) for the nodes in the database. Use IP addresses to avoid dependence on DNS lookups. Host names must be absolute. You cannot use `localhost` or `127.0.0.1` as a host name. See Specifying Hosts for more information. | None
--package=name -k | Name of the HADB package (version). If the package is not found, a default package is registered. This option is deprecated. Use the hadbm registerpackage command to register a package in the domain. | None
--packagepath=path-L | Path to the HADB software package. Use only if the package is not registered in the domain. This option is deprecated. Use the hadbm registerpackage command to register a package in the domain. | None
--portbase= number -b | Port base number used for node 0. Successive nodes are automatically assigned port base numbers in steps of 20 from this number. Each node uses its port base number and the next five consecutively numbered ports. To run several databases on the same machine, have a plan for allocating port numbers explicitly. | 15200
--spares= number -s | Number of spare nodes. This number must be even and must be less than the number of nodes specified in the `--hosts` option. | 0
--set=attr-val-list -S | Comma-separated list of database configuration attributes in name =value format. For explanations of database configuration attributes, see Clearing and Archiving History Files | None

##### Example 3–3 Example of creating a database

The following command is an example of creating a database:

    hadbm create --spares 2 --devicesize 1024 --dbpassword secret123
     --hosts n0,n1,n2,n3,n4,n5

### Specifying Hosts

Use the `--hosts` option to specify a comma-separated list of host names or IP addresses for the nodes in the database. The `hadbm create` command creates one node for each host name (or IP address) in the list. The number of nodes must be even. Use duplicate host names to create multiple nodes on the same machine with different port numbers. Make sure that nodes on the same machine are not mirror nodes.

Nodes are numbered starting at zero in the order listed in this option. The first mirrored pair are nodes zero (0) and one (1), the second two (2) and three (3), and so on. Odd numbered nodes are in one DRU, even numbered nodes in the other. With `--spares` option, spare nodes are those with the highest numbers.

For information about configuring double network interfaces, see Configuring Network Redundancy

### Specifying Device Size

Specify the device size using the `--devicesize` option. The recommended device size is:

    (4x / nd + 4l/d) / 0.99

Where

- x is the total size of user data

- n is the number of nodes (given by the `--hosts` option)

- d is the number of devices per node (given by the `--datadevices` option)

- l is the log buffer size (given by the attribute `LogBufferSize`)

  If refragmentation might occur (for example, using `hadbm addnodes`), then the recommended device size is:

      (8x / nd + 4l/d) / 0.99

### Setting Heterogeneous Device Paths

To set a different device path for each node or service, use the `--` `set` option of `hadbm create`. There are four types of devices: the `DataDevice`, the `NiLogDevice` (node internal log device), the `RelalgDevice` (relational algebra query device), and the `NoManDevice` (node manager device). The syntax for each name =value pair is as follows, where `-`devno is required only if the device is `DataDevice`:

`node-nodeno.device-devno.Devicepath`

For example:

    --set Node-0.DataDevice-0.DevicePath=/disk0,
    Node-1.DataDevice-0.DevicePath=/disk 1

You can also set a heterogeneous path to history files, as follows:

    node-nodeno.historypath=path

For information on history files, see Clearing and Archiving History Files

Any device path that is not set for a particular node or device defaults to the `--devicepath` value.

Note –

Change device paths and location of history files using `hadbm set` and `hadbm addnodes` commands.

### Troubleshooting

If you have difficulty creating a database, check the following:

- Ensure you have started the management agents on all the hosts and defined an HADB domain. For details, see Starting the Management Agent

- File and directory permissions must be set to allow read, write, and execute access to the install, history, device, and config paths for the following users:

  - Sun Java System Application Server administrative user (set during installation)

  - HADB system user

    For details about setting user permissions, see Preparing for HADB Setup

  Application Server and HADB port assignments must not conflict with other port assignments on the same machine. Default recommended port assignments are:

  - Sun Java SystemMessage Queue: 7676

  - IIOP: 3700

  - HTTP server: 80

  - Administration server: 4848

  - HADB nodes: Each node uses six consecutive ports. For example, for default port 15200, node 0 uses 15200 through 15205 , node 1 uses 15220 through 15225, and so on.

  Disk space must be adequate; see the Sun Java System Application Server Release Notes.

## Viewing and Modifying Configuration Attributes

You can view and modify database configuration attributes with the `hadbm get and hadbm set` commands, respectively.

### Getting the Values of Configuration Attributes

To get the values of configuration attributes, use the `hadbm get` command. For a list of valid attributes, see Configuration Attributes. The command syntax is:

    hadbm get attribute-list | --all
    [dbname]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]

The dbname operand specifies the database name. The default is `hadb`.

The attribute-list operand is a comma-separated or quote-enclosed space-separated list of attributes. The `--all` option displays values for all attributes. For a list of all attributes for `hadbm get`, see Configuration Attributes.

See General Options for a description of command options.

##### Example 3–4 Example of using hadbm get

    hadbm get JdbcUrl,NumberOfSessions

### Setting the Values of Configuration Attributes

To set the values of configuration attributes, use the `hadbm set` command. For a list of valid attributes, see Configuration Attributes

    hadbm set [dbname] attribute
    =value[,attribute=
    value...]
     [--adminpassword=password | --adminpasswordfile=file]
     [--agent=maurl]

The dbname operand specifies the database name. The default is `hadb`.

The attribute=value list is a comma-separated or quote-enclosed space-separated list of attributes.

See General Options for a description of command options.

If this command executes successfully, it restarts the database in the state it was in previously, or in a better state. For information about database states, see Getting the Status of HADB restart the HADB as described in Restarting a Database.

You cannot set the following attributes with `hadbm set`. Instead, set them when you create a database (see Creating a Database).

- DatabaseName

- DevicePath

- HistoryPath

- NumberOfDatadevices

- Portbase

- JdbcUrl (its value is set during database creation based on the `--hosts` and `--portbase` options).

Note –

Using `hadbm set` to set any configuration attribute, except `ConnectionTrace` or `SQLTraceMode`, causes a rolling restart of HADB. In a rolling restart, each node is stopped, and started with the new configuration, one at a time; HADB services are not interrupted.

If you set `ConnectionTrace` or `SQLTraceMode`, no rolling restart occurs, but the change only takes effect for new HADB connections made from an Application Server instance.

### Configuration Attributes

The following table lists the configuration attributes that you can modify with `hadbm set` and retrieve with `hadbm get`.

Table 3–8 Configuration Attributes

Attribute | Description | Default | Range
--- | --- | --- | ---
ConnectionTrace | If true, records a message in the HADB history files when a client connection (JDBC, ODBC) is initiated or terminated. | False | True or False
CoreFile | Do not change the default value. | False | True or False
DatabaseName | Name of the database. | hadb |
DataBufferPoolSize | Size of the data buffer pool allocated in shared memory. | 200MB | 16 - 2047 MB
DataDeviceSize | Specifies the device size for the node. For information on the recommended `DataDeviceSize`, see Specifying Device Size The maximum value is the smaller of 256GB or the maximum operating system file size. The minimum value is: (4 x LogbufferSize + 16MB) / n where n is number of data devices. | 1024MB | 32 - 262144 MB
PackageName | Name of HADB software package used by the database. | V4.x.x.x | None
DevicePath | Location of the devices. Devices are: Data device (`DataDevice`) Node internal log device (`NiLogDevice`) Relational algebra query device (`RelalgDevice`) | Solaris and Linux: `/var/opt/SUNWhadb` Windows: `C:\Sun\AppServer\SUNWhadb\vers`, where vers is the HADB version number. |
EagerSessionThreshold | Determines whether normal or eager idle session expiration is used. In normal idle session expiration, sessions that are idle for more than `SessionTimeout` seconds are expired. When the number of concurrent sessions exceeds the `EagerSessionThreshold` percentage of the maximum number of sessions, sessions that are idle for more than `EagerSessionTimeout` seconds are expired. | Half of NumberOfSessions attribute | 0 - 100
EagerSessionTimeout | The time in seconds a database connection can be idle before it expires when eager session expiration is used. | 120 seconds | 0-2147483647 seconds
EventBufferSize | Size of the event buffer, where database events are logged. If set to `0`, no event buffer logging is performed. During failures, the event buffer is dumped. This gives valuable information on the cause of the failures and is useful during trial deployment. Writing events to memory has a performance penalty. | `0 MB` | 0-2097152 MB
HistoryPath | Location of the HADB history files, which contain information, warnings, and error messages. This is a read-only attribute. | Solaris and Linux: `/var/opt/SUNWhadb` Windows: REPLACEDIR (replaced by the actual URL at runtime.) |
InternalLogbufferSize | Size of the node internal log device, which keeps track of operations related to storing data. | 12MB | 4 - 128 MB
JdbcUrl | The JDBC connection URL for the database. This is a read-only attribute. | none |
LogbufferSize | Size of the log buffer, which keeps track of operations related to data. | 48MB | 4 - 2048 MB
MaxTables | Maximum number of tables allowed in an HADB database. | 1100 | 100 - 1100
NumberOfDatadevices | Number of data devices used by an HADB node. This is a read-only attribute. | 1 | 1 - 8
NumberOfLocks | Number of locks allocated by an HADB node. | 50000 | 20000-1073741824
NumberOfSessions | Maximum number of sessions (database connections) that can be opened for an HADB node. | 100 | 1 - 10000
PortBase | Base port number used to create different port numbers for different HADB processes. This is a read-only attribute. | 15200 | 10000 - 63000
RelalgDeviceSize | Size of the device used in relational algebra queries. | 128 MB | 32 - 262144 MB
SessionTimeout | Amount of time a database connection can be idle before it expires when normal session expiration is used. | 1800 seconds | 0-2147483647 seconds
SQLTraceMode | Amount of information about executed SQL queries written to the history files. If `SHORT`, login and logout of SQL sessions are recorded. If `FULL`, all SQL queries being prepared and executed, including parameter values, are recorded. | NONE | NONE/SHORT/ FULL
StartRepairDelay | Maximum time a spare node allows for a failed active node to perform a node recovery. If the failed node cannot recover within this time interval, the spare node starts copying data from the failed node’s mirror and becomes active. Changing the default value is not recommended. | 20 seconds | 0 - 100000 seconds
StatInterval | Interval at which an HADB node writes throughput and response time statistics to its history file. To disable, set to 0. Here is an example of a statistics line: `Req-reply time: # 123, min= 69 avg= 1160 max= 9311 %=100.0` The number after the has sign (`#)` is the number of requests serviced over the `StatInterval`. The next three numbers are the minimum, average, and maximum time in microseconds taken by transactions completed over the `StatInterval`. The number afer the percent sign ( `%)` is the number of transactions completed successfully within 15 milliseconds over the `StatInterval`. | 600 seconds | 0 - 600 seconds
SyslogFacility | Facility used when reporting to `syslog`. The `syslog` daemon should be configured (see `man syslogd.conf` for details). Use a facility that is not used by other applications running on the same machine. Set to `none` to disable `syslog` logging. | local0 | local0, local1, local2, local3, local4, local5, local6, local7, kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, none
SysLogging | If true, an HADB node writes information to the operating system’s `syslog` files. | True | True or False
SysLogLevel | Minimum level of HADB message saved to operating system’s `syslog` files. All messages of that level or higher will be logged. For example, “info” logs all messages. | warning | nonealert errorwarninginfo
SyslogPrefix | Text string inserted before all `syslog` messages written by the HADB. | hadb -dbname |
TakeoverTime | Time between when a node fails and when its mirror takes over. Do not change the default value. | 10000 (milliseconds) | 500 - 16000 milliseconds

## Configuring the JDBC Connection Pool

Application Server communicates with HADB using the Java Database Connectivity (JDBC) API. The `asadmin configure-ha-cluster` command automatically creates a JDBC connection pool for use with HADB (for a cluster cluster-name ). The name of the connection pool is `cluster-name-hadb-pool`. The JNDI URL of JDBC resource is `jdbc/cluster-name-hastore`.

The initial configuration of the connection pool is normally sufficient. When you add a node, change the steady pool size so that there are eight connections for each HADB active node. See Adding Nodes.

This chapter covers the following topics:

- Getting the JDBC URL

- Creating a Connection Pool

- Example 3–5

- Creating a JDBC Resource

  For general information about connection pools and JDBC resources, see Administration Guide.

### Getting the JDBC URL

Before you can set up the JDBC connection pool, you need to determine the JDBC URL of HADB using the `hadbm get` command as follows:

    hadbm get JdbcUrl [dbname]

For example:

    hadbm get JdbcUrl

This command displays the JDBC URL, which is of he following form:

    jdbc:sun:hadb:host:port,
    host:port,...

Remove the `jdbc:sun:hadb:` prefix and use the host`:`port`,` host`:`port`...` part as the value of the `serverList` connection pool property, described in Table 3–10.

### Creating a Connection Pool

The following table summarizes connection pool settings required for the HADB. Change the Steady Pool Size when adding nodes, but do not change other settings.

Table 3–9 HADB Connection Pool Settings

Setting | Required Value for HADB
--- | ---
Name | The HADB JDBC resource’s Pool Name setting must refer to this name
Database Vendor | HADB 4.4
Global Transaction Support | Unchecked/false
DataSource Classname |
    com.sun.hadb.jdbc.ds.HadbDataSource
Steady Pool Size | Use 8 connections for each active HADB node. For more detailed information, see the System Deployment Guide.
Connection Validation Required | Checked/true
Validation Method | `meta-data`
Table Name | Do not specify
Fail All Connections | Unchecked/false
Transaction Isolation | `repeatable-read`
Guarantee Isolation Level | Checked/true

The following table summarizes connection pool properties required for the HADB. Change `serverList` when adding nodes, but do not change other properties.

Table 3–10 HADB Connection Pool Properties

Property | Description
--- | ---
username | Name of the `storeuser` to use in the `asadmin create-session-store` command.
password | Password (`storepassword` ) to us in the `asadmin create-session-store` command.
serverList | JDBC URL of the HADB. To determine this value, see Getting the JDBC URL You must change this value if you add nodes to the database. See Adding Nodes.
cacheDatabaseMetaData | When `false` , as required, ensures that calls to `Connection.getMetaData()` make calls to the database, which ensures that the connection is valid.
eliminateRedundantEndTransaction | When `true` , as required, improves performance by eliminating redundant commit and rollback requests and ignoring these requests if no transaction is open.
maxStatement | Maximum number of statements per open connection that are cached in the driver statement pool. Set this property to `20`.

##### Example 3–5 Creating a Connection Pool

Here is an example `asadmin create-jdbc-connection-pool` command that creates an HADB JDBC connection pool:

    asadmin create-jdbc-connection-pool
    --user adminname --password secret
    --datasourceclassname com.sun.hadb.jdbc.ds.HadbDataSource
    --steadypoolsize=32
    --isolationlevel=repeatable-read
    --isconnectvalidatereq=true
    --validationmethod=meta-data
    --property username=storename:password=secret456:serverList=
    host\:port,host\:port,
    host\\:port,host\:port,
    host\:port,host\:port
    :cacheDatabaseMetaData=false:eliminateRedundantEndTransaction=true hadbpool

On Solaris, escape colon characters (`:`) within property values with double backslashes (`\\`). On Windows, escape colon characters (`:`) with single backslashes ( `\`).

### Creating a JDBC Resource

The following table summarizes JDBC resource settings required for HADB.

Table 3–11 HADB JDBC Resource Settings

Setting | Description
--- | ---
JNDI Name | The following JNDI name is the default in the session persistence configuration: `jdbc/hastore`. You can use the default name or a different name. You must also specify this JNDI name as the value of the `store-pool-jndi-name` Persistence Store property when you activate the availability service.
Pool Name | Select from the list the name (or ID) of the HADB connection pool used by this JDBC resource. For more information, see Configuring Network Redundancy
Data Source Enabled | Checked/true

# Managing HADB

You generally need to perform management operations when you replace or upgrade your network, hardware, operating system, or HADB software. The following sections explain various management operations:

- Managing Domains

- Managing Nodes

- Managing Databases

- Recovering from Session Data Corruption

## Managing Domains

You can perform the following operations on an HADB domain:

- Creating a Domain: For more information, see Creating a Management Domain

- Extending a Domain

- Deleting a Domain

- Listing Hosts in a Domain

- Removing Hosts from a Domain

See Security Options and General Options for a description of command options.

### Extending a Domain

Use `extenddomain` to add hosts to an existing management domain. The command syntax is:

    hadbm extenddomain
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    hostlist

IP addresses of HADB hosts must be IPv4 addresses.

For more information, see hadbm-extenddomain(1).

### Deleting a Domain

Use `deletedomain to` remove a management domain. The command syntax is:

    hadbm deletedomain
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]

For more information, see hadbm-deletedomain(1).

### Removing Hosts from a Domain

Use `reducedomain` to remove hosts from the management domain. The command syntax is:

    hadbm reducedomain
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    host_list

For more information, see hadbm-reducedomain(1).

### Listing Hosts in a Domain

Use `listdomain` to list all hosts defined in the management domain. The command syntax is:

    hadbm listdomain
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]

For more information, see hadbm-listdomain(1).

## Managing Nodes

You can perform the following operations on individual nodes:

- Starting a Node

- Stopping a Node

- Restarting a Node

### Starting a Node

You might need to manually start an HADB node that was stopped because its host was taken off-line for a hardware or software upgrade or replacement. Also, you might need to manually start a node if it fails to restart for some reason (other than a double failure). For more information on how to recover from double failures, see Clearing a database.

In most cases, you should first attempt to start the node using the `normal` start level. You must use the `repair` start level if the `normal` start level fails or times out.

To start a node in the database, use the `hadbm startnode` command. The syntax is:

    hadbm startnode
     [--adminpassword=password | --adminpasswordfile=file]
     [--agent=maurl]
     [--startlevel=level]
     nodeno
     [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The nodeno operand specifies the number of the node to start. Use `hadbm status` to display the numbers of all nodes in a database.

For more information, see hadbm-startnode(1).

#### Start level option

The `hadbm startnode` command has one special option, `--startlevel` (short form `-l`), that specifies the level at which to start the node.

Node start levels are:

- normal (default): starts the node with the data found locally on the node (in the memory and in the data device file on the disk) and synchronizes it with the mirror for recent updates it missed.

- repair: forces the node to discard local data and copy it from its mirror.

- clear: reinitializes the devices for the node and forces a repair of data from its mirror node. Use when the device files need to be initialized, necessary if they are damaged or the disk that contained the device files is replaced.

See General Options for a description of other command options.

##### Example 3–6 Example of starting a node

    hadbm startnode 1

### Stopping a Node

You might need to stop a node to repair or upgrade the host machine’s hardware or software. To stop a node, use the `hadbm stopnode` command. The command syntax is:

    hadbm stopnode
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [--no-repair]
    nodeno
    [dbname]

The nodeno operand specifies the number of the node to stop. The mirror node of this node number must be running. Use `hadbm status` to display the numbers of all nodes in a database.

The dbname operand specifies the database name. The default is `hadb`.

The `hadbm stopnode` command has one special option, `--no-repair` (short form `-R`) that indicates no spare node is to replace the stopped node. Without this option, a spare node starts up and takes over the functioning of the stopped node.

See General Options for a description of other command options. For more information, see hadbm-stopnode(1).

##### Example 3–7 Example of stopping a node

    hadbm stopnode 1

### Restarting a Node

You might have to restart a node if you notice unusual behavior such as excessive CPU consumption.

To restart a node in the database, use the `hadbm restartnode` command. The command syntax is:

    hadbm restartnode
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [--startlevel=level]
    nodeno
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The nodeno operand specifies the number of the node to restart. Use `hadbm status` to display the numbers of all nodes in a database.

The `hadbm restartnode` command has one special option, `--startlevel` (short form `-l`), that specifies the level at which to start the node. See Start level option for more information.

See General Options for a description of other command options. For more information, see hadbm-restartnode(1).

##### Example 3–8 Example of restarting a node

    hadbm restartnode 1

## Managing Databases

You can perform the following operations on HADB databases:

- Starting a Database

- Stopping a Database

- Restarting a Database

- Listing Databases

- Clearing a database

- Removing a Database

### Starting a Database

To start a database, use the `hadbm start` command. This command starts all nodes that were running before the database was stopped. Individually stopped (offline) nodes are not started when the database is started after a stop.

The command syntax is:

    hadbm start
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

See General Options for a description of command options. For more information, see hadbm-start(1).

##### Example 3–9 Example of starting a database

    hadbm start

### Stopping a Database

When you stop and start a database in separate operations, data is unavailable while it is stopped. To keep data available, you can restart a database as described in Restarting a Database.

Stop a database to:

- Remove the database.

- Perform system maintenance that affects all HADB nodes.

Before stopping a database, either stop dependent Application Server instances that are using the database, or configure them to use a Persistence Type other than `ha`.

When you stop the database, all the running nodes in the database are stopped and the status of the database becomes Stopped. For more information about database states, see Getting the Status of HADB.

To stop a database, use the `hadbm stop` command. The command syntax is:

    hadbm stop
    [--adminpassword=password | --adminpasswordfile= file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

See General Options for a description of command options. For more information, see hadbm-stop(1).

##### Example 3–10 Example of stopping a database

    hadbm stop

### Restarting a Database

You might want to restart a database if you notice strange behavior (for example consistent timeout problems). In some cases, a restart may solve the problem.

When you restart a database, y, the database and its data remain available. When you stop and start HADB in separate operations, data and database services are unavailable while HADB is stopped. This is because by default `hadbm restart` performs a rolling restart of nodes: it stops and starts the nodes one by one. In contrast, `hadbm stop` stops all nodes simultaneously.

To restart a database, use the `hadbm restart` command. The command syntax is:

    hadbm restart
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [--no-rolling]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

This command has one special option, `--no-rolling` (short form `-g`), that specifies to restart all nodes at once, which causes loss of service. Without this option, this command restarts each of the nodes in the database to the current state or a better state.

See General Options for a description of other command options. For more information, see hadbm-restart(1).

For example:

    hadbm restart

### Listing Databases

To list all the databases in an HADB instance, use the `hadbm list` command. The command syntax is:

    hadbm list
    [--agent=maurl]
    [--adminpassword=password | --adminpasswordfile=file]

See General Options for a description of command options. For more information, see hadbm-list(1).

### Clearing a database

Clear a database when:

- The `hadbm status` command reveals that the database is non-operational or if See Getting the Status of HADB.

- Multiple nodes do not respond and are in waiting state for a long time.

- Recovering from session data corruption. See Recovering from Session Data Corruption

The `hadbm clear` command stops the database nodes, clears the database devices, then starts the nodes. This command erases the Application Server schema data store in HADB, including tables, user names, and passwords. After running `hadbm clear`, use `asadmin configure-ha-cluster` to recreate the data schema, reconfigure the JDBC connection pool, and reload the session persistence store.

The command syntax is:

    hadbm clear  [--fast]  [--spares=number]
    [--dbpassword=password | --dbpasswordfile= file]
    [--adminpassword=password | --adminpasswordfile= file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The following table describes the special `hadbm clear` command options. See General Options for a description of other options.

For more information, see hadbm-clear(1).

Table 3–12 `hadbm clear` Options

Option | Description | Default
--- | --- | ---
--fast -F | Skips device initialization while initializing the database. Do not use if the disk storage device is corrupted. | Not applicable
--spares= number -s | Number of spare nodes the reinitialized database will have. Must be even and less than the number of nodes in the database. | Previous number of spares

For example:

    hadbm clear --fast --spares=2 --dbpassword secret123

### Removing a Database

To remove an existing database, use the `hadbm delete` command. This command deletes the database’s configuration files, device files, and history files, and frees shared memory resources. The database you want to remove must exist and must be stopped. See Stopping a Database.

The command syntax is:

    hadbm delete
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

See General Options for a description of command options. For more information, see hadbm-delete(1).

##### Example 3–11 Example of removing a database

The command:

    hadbm delete

deletes the default database, `hadb`.

## Recovering from Session Data Corruption

The following are indications that session data may be corrupted:

- Error messages appear in the Application Server system log (`server.log`) every time an application tries to save session state.

- Error messages in the server log indicate that the session could not be found or could not be loaded during session activation.

- Sessions that are activated after previously being passivated contain empty or incorrect session data.

- When an instance fails, failed-over sessions contain empty or incorrect session data.

- When an instance fails, instances that try to load a failed-over session cause an error in the server log indicating the session could not be found or could not be loaded.

### To bring the session store back to a consistent state

If you determine that the session store has been corrupted, bring it back to a consistent state by following this procedure:

1. Clear the session store.

   Determine if this action corrects the problem. If it does, then stop. If not—for example, if you continue to see errors in the server log—then continue.

2. Re-initialize the data space on all the nodes and clear the data in the database.

   See Clearing a database .

   Determine if this action corrects the problem. If it does, then stop. If not—for example, if you continue to see errors in the server log—then continue.

3. Delete and then recreate the database.

   See Removing a Database and Creating a Database

# Expanding HADB

There are two reasons to expand your original HADB configuration:

- Volume of session data being saved increases beyond existing storage space in data devices. Transactions may start aborting due to full data devices.

- User load increases, exhausting system resources. You need to add more hosts.

This section describes how you can expand HADB without shutting down your Application Server cluster or database, in particular:

- Adding Storage Space to Existing Nodes

- Adding Machines

- Adding Nodes

- Refragmenting the Database

- Adding Nodes by Recreating the Database

Also see related information in Maintaining HADB Machines .

## Adding Storage Space to Existing Nodes

Add HADB storage space:

- If user transactions repeatedly abort with one of the following error messages:

  - `4592: No free blocks on data devices`

  - `4593: No unreserved blocks on data devices`

- If the `hadbm deviceinfo` command consistently reports insufficient free size. See Getting Device Information.

You may also want to add storage space to existing nodes if there is unused disk space on the nodes or when you add disk capacity. For information on the recommended data device size, see Specifying Device Size

To add storage space to nodes, use the `hadbm set` command to increase data device size.

The command syntax is:

    hadbm set DataDeviceSize=size

where size is the data device size in MBytes.

See General Options for a description of command options.

Changing the data device size for a database in a `FaultTolerant` or higher state upgrades the system without loss of data or availability. The database remains in operational during the reconfiguration. Changing device size on a system that is not `FaultTolerant` or better causes loss of data. For more information about database states, see Database States.

##### Example 3–12 Example of setting data device size

The following command is an example of setting data device size:

    hadbm set DataDeviceSize=1024

## Adding Machines

You may want to add machines if HADB requires more processing or storage capacity. To add a new machine on which to run HADB, install HADB packages with or without the Application Serveras described in Chapter 2, Installing and Setting Up High Availability Database. For an explanation of node topology alternatives, see Chapter 3, Selecting a Topology, in Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Deployment Planning Guide.

### To add new machines to an existing HADB instance

1. Start management agents on the new nodes.

2. Extend the management domain to the new hosts.

   For details, see `hadbm extenddomain` command.

3. Start the new nodes on these hosts.

   For details, see Adding Nodes

## Adding Nodes

To increase processing and storage capacity of an HADB system, create new nodes and add them to the database.

After you add nodes, update the following properties of the HADB JDBC connection pool:

- The `serverlist` property.

- Steady pool size. Generally, you add 8 more connections for each new node. For more information, see System Sizing in Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Deployment Planning Guide.

To add nodes, use the `hadbm addnodes` command. The command syntax is:

    hadbm addnodes  [--no-refragment]  [--spares=sparecount]
    [--historypath=path]
    [--devicepath=path]
    [--set=attr-name-value-list]
    [--dbpassword=password | --dbpasswordfile=file ]
    [--adminpassword=password | --adminpasswordfile=file]
    --hosts=hostlist  [dbname]

The dbname operand specifies the database name. The default is `hadb`. The database must be in `HAFaultTolerant` or `FaultTolerant` state. For more information about database states, see Getting the Status of HADB.

If you do not specify the `--devicepath` and `--historypath` options, the new nodes will have the same device path and use the same history files as the existing database.

Adding nodes performs a refragmentation and redistribution of the existing data to include the new nodes in the system. Online refragmenting requires that the disks for the HADB nodes have enough space to contain the old data and the new data simultaneously until refragmenting is finished, that is, the user data size must not exceed 50% of the space available for user data. For details, see Getting Device Information

Note –

The best time to add nodes is when the system is lightly loaded.

##### Example 3–13 Example of adding nodes

For example:

    hadbm addnodes --dbpassword secret123 -adminpassword=
    password --hosts n6,n7,n8,n9

The following table describes the special `hadbm addnodes` command options. See General Options for a description of other options.

Table 3–13 `hadbm addnodes` Options

Option | Description | Default
--- | --- | ---
--no-refragment -r | Do not refragment the database during node creation; In this case, refragment the database later using the `hadbm refragment` command to use the new nodes. For details about refragmentation, see Refragmenting the Database If you do not have sufficient device space for refragmentation, recreate the database with more nodes. See Adding Nodes by Recreating the Database | Not applicable
--spares= number -s | Number of new spare nodes in addition to those that already exist. Must be even and not greater than the number of nodes added. | 0
--devicepath= path -d | Path to the devices. Devices are: `DataDevice` `NiLogDevice` (node internal log device) `RelalgDevice` (relational algebra query device) This path must already exist and be writable. To set this path differently for each node or each device, see Setting Heterogeneous Device Paths | Solaris and Linux: `HADB_install_dir/device` Windows: `C:\Sun\AppServer\SUNWhadb\vers`, where vers is the HADB version number.
--hosts= hostlist -H | Comma-separated list of new host names for the new nodes in the database. One node is created for each comma-separated item in the list. The number of nodes must be even. IP addresses of HADB hosts must be IPv4 addresses. Using duplicate host names creates multiple nodes on the same machine with different port numbers. Make sure that nodes on the same machine are not mirror nodes. Odd numbered nodes are in one DRU, even numbered nodes in the other. If `--spares` is used, new spare nodes are those with the highest numbers. If the database was created with double network interfaces, the new nodes must be configured in the same way. See Configuring Network Redundancy . | None

## Refragmenting the Database

Refragment the database to store data in newly-created nodes. Refragmentation distributes data evenly across all active nodes.

To refragment the database, use the `hadbm refragment` command. The command syntax is:

    hadbm refragment  [--dbpassword=password | --dbpasswordfile=file]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`. The database must be in `HAFaultTolerant` or `FaultTolerant` state. For more information about database states, see Getting the Status of HADB.

See General Options for a description of command options. For more information, see hadbm-refragment(1).

Online refragmentation requires that the disks for the HADB nodes have enough space to contain the old data and the new data simultaneously until refragmenting is finished, that is, the user data size must not exceed 50% of the space available for user data. For details, see Getting Device Information

Note –

The best time to refragment the database is when the system is lightly loaded.

If this command fails after multiple attempts, see Adding Nodes by Recreating the Database

##### Example 3–14 Example of refragmenting the database

For example:

    hadbm refragment --dbpassword secret123

## Adding Nodes by Recreating the Database

If online refragmentation fails persistently when you add new nodes (either due to lack of data device space or other reasons), recreate the database with new nodes. This will lead to the loss of existing user data and schema data.

### To add nodes by recreating the database

This procedure enables you to maintain HADB availability throughout the process.

1. For each Application Server instance:

   1. Disable the Application Server instance in the load balancer.

   2. Disable session persistence.

   3. Restart the Application Server instance.

   4. Re-enable the Application Server instance in the load balancer.

   If you do not need to maintain availability, you can disable and re-enable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.

2. Stop the database as described in Stopping a Database .

3. Delete the database as described in Removing a Database .

4. Recreate the database with the additional nodes as described in Creating a Database

5. Reconfigure the JDBC connection pool as described in Configuring the JDBC Connection Pool

6. Reload the session persistence store.

7. For each Application Server instance:

   1. Disable the Application Server instance in the load balancer.

   2. Enable session persistence.

   3. Restart the Application Server instance.

   4. Re-enable the Application Server instance in the load balancer.

   If you do not need to maintain availability, you can disable and re-enable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.

# Monitoring HADB

You can monitor the activities of HADB by:

- Getting the Status of HADB

- Getting Device Information

- Getting Runtime Resource Information

These sections briefly describe the `hadbm status`, `hadbm deviceinfo`, and `hadbm resourceinfo` commands. For information on interpreting HADB information, see Performance in Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Performance Tuning Guide.

## Getting the Status of HADB

Use the `hadbm status` command to display the status of the database or its nodes. The command syntax is:

    hadbm status
    [--nodes]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The `--nodes` option (short form `-n`) displays information on each node in the database. For more information , see Node Status. See General Options for a description of other command options.

For more information, see hadbm-status(1).

##### Example 3–15 Example of getting HADB status

For example:

    hadbm status --nodes

### Database States

A database’s state summarizes its current condition. The following table describes the possible database states.

Table 3–14 HADB States

Database State | Description
--- | ---
High-Availability Fault Tolerant (HAFaultTolerant) | Database is fault tolerant and has at least one spare node on each DRU.
Fault Tolerant | All the mirrored node pairs are up and running.
Operational | At least one node in each mirrored node pair is running.
Non Operational | One or more mirrored node pairs is missing both nodes. If the database is non-operational, clear the database as described in Clearing a database.
Stopped | No nodes are running in the database.
Unknown | Cannot determine the state of the database.

### Node Status

Use the`--nodes` option to make the `hadbm status` command display the following information for each node in the database:

- Node number

- Name of the machine where the node is running

- Port number of the node

- Role of the node. For a list of roles and their meanings, see Roles of a Node

- State of the node. For a list of states and their meanings, see States of a Node

- Number of the corresponding mirror node.

A node’s role and state can change as described in these sections:

- Roles of a Node

- States of a Node

#### Roles of a Node

A node is assigned a role during its creation and can take any one of these roles:

- Active: Stores data and allows client access. Active nodes are in mirrored pairs.

- Spare: Allows client access, but does not store data. After initializing data devices, monitors other data nodes to initiate repair if another node becomes unavailable.

- Offline: Provide no services until their role changes. When placed back online, its role can change back to its former role.

- Shutdown: An intermediate step between active and offline, waiting for a spare node to take over its functioning. After the spare node has taken over, the node is taken offline.

#### States of a Node

A node can be in any one of the following states:

- Starting: The node is starting.

- Waiting: The node cannot decide its start level and is offline. If a single node is in this state for more than two minutes, stop the node and then start it at the `repair` level; see Stopping a Node and Starting a Node Clearing a database.

- Running: The node is providing all services that are appropriate for its role.

- Stopping: The node is in the process of stopping.

- Stopped: The node is inactive. Repair of a stopped node is prohibited.

- Recovering: The node is being recovered. When a node fails, the mirror node takes over the functions of the failed node. The failed node tries to recover by using the data and log records in main memory or on disk. The failed node uses the log records from the mirror node to catch up with the transactions performed when it was down. If recovery is successful, the node becomes active. If recovery fails, the node state changes to repairing.

- Repairing: The node is being repaired. This operation reinitializes the node and copies the data and log records from the mirror node. Repair is more time consuming than recovery.

## Getting Device Information

Monitor free space in HADB data (disk storage) devices:

- Routinely, to check the trend in disk space use.

- As part of preventive maintenance: if the user load has increased and you want to resize or scale the database configuration.

- As part of scaling up the database: Before running `hadbm addnodes` to add new nodes to the system, check whether there is enough device space. Remember, you need around 40-50% free space on the existing nodes to add nodes.

- When you see messages in the history files and `server.log` file such as

  - `No free blocks on data devices`

  - `No unreserved blocks on data devices` .

Use the `hadbm deviceinfo` command to get information about free space in data devices. This command displays the following information for each node of the database:

- Total device size allocated, in MB (Totalsize).

- Free space in MB (Freesize).

- Percent of device currently being used (Usage)

The command syntax is:

    hadbm deviceinfo  [--details]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]  [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The `--details` option displays the following additional information:

- Number of read operations by the device.

- Number of write operations by the device.

- Name of the device.

See General Options for a description of other command options.

For more information, see hadbm-deviceinfo(1).

To determine the space available for user data, take the total device size, then subtract the space reserved for HADB: four times the `LogBufferSize` `+ 1%` of the device size. If you do not know the size of the log buffer, use the command `hadbm get logbufferSize`. For example, if the total device size is 128 MB and the `LogBufferSize` is 24 MB, the space available for user data is 128 – (4 x 24) = 32 MB. Of the 32 MB, half is used for replicated data and around one percent is used for the indices, and only 25 percent is available for the real user data.

The space available for user data is the difference between the total size and reserved size. If the data is refragmented in the future, the free size must be approximately equal to 50% of the space available for user data. If refragmentation is not relevant, the data devices can be exploited to their maximum. Resource consumption warnings are written to the history files when the system is running short on device space.

For more information about tuning HADB, see the Sun Java System Application Server Performance Tuning Guide.

##### Example 3–16 Example of getting device information

The following command:

    hadbm deviceinfo --details

Displays the following example results:

    NodeNO Totalsize Freesize Usage NReads NWrites DeviceName
    0      128       120      6%    10000  5000    C:\Sun\SUNWhadb\hadb.data.0
    1      128       124      3%    10000  5000    C:\Sun\SUNWhadb\hadb.data.1
    2      128       126      2%     9500  4500    C:\Sun\SUNWhadb\hadb.data.2
    3      128       126      2%     9500  4500    C:\Sun\SUNWhadb\hadb.data.3

## Getting Runtime Resource Information

The `hadbm resourceinfo` command displays HADB runtime resource information. You can use this information to help identify resource contention, and reduce performance bottlenecks. For details, see Tuning HADB in Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Performance Tuning Guide.

The command syntax is:

    hadbm resourceinfo  [--databuf]  [--locks]  [--logbuf]  [--nilogbuf]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]
    [dbname]

The dbname operand specifies the database name. The default is `hadb`.

The following table describes the `hadbm resourceinfo` special command options. See General Options for a description of other command options.

For more information, see hadbm-resourceinfo(1).

Table 3–15 hadbm resourceinfo Command Options

Option | Description
--- | ---
--databuf -d | Display data buffer pool information. See Data Buffer Pool Information below for more information.
--locks -l | Display lock information. See Lock Information below for more information.
--logbuf -b | Display log buffer information. See Log Buffer Information below for more information.
--nilogbuf -n | Display node internal log buffer information. See Node Internal Log Buffer Information below for more information.

### Data Buffer Pool Information

Data buffer pool information contains the following:

- `NodeNo`: Node number.

- `Avail`: Total space available in the pool, in MBytes.

- `Free`: Free space available, in MBytes.

- `Access`: Cumulative number of accesses to the data buffer from database, from start until now.

- `Misses`: Cumulative number of page faults that have occurred from database start until now.

- `Copy-on-Write`: Cumulative number of pages copied internally in the data buffer due to checkpointing.

When a user transaction performs an operation on a record, the page containing the record must be in the data buffer pool. If it is not, a `miss` or a page fault occurs. The transaction then has to wait until the page is retrieved from the data device file on the disk.

If the miss rate is high, increase the data buffer pool. Since the misses are cumulative, run `hadbm resourceinfo` periodically and use the difference between two runs to see the trend of miss rate. Do not be concerned if free space is very small, since the checkpointing mechanism will make new blocks available.

##### Example 3–17 Example data buffer pool information

For example:

    NodeNO Avail Free Access Misses Copy-on-Write
    0 256 128 100000 50000 10001 256 128 110000 45000 950

### Lock Information

Lock information is as follows:

- `NodeNo`: Node Number.

- `Avail`: Total number of locks available on the node.

- `Free`: Number of free locks.

- `Waits`: Number of transactions waiting to acquire locks. This is cumulative.

One single transaction cannot use more than 25% of the available locks on a node. Therefore, transactions performing operations in large scale should be aware of this limitation. It is best to perform such transactions in batches, where each batch must be treated as a separate transaction, that is, each batch commits. This is needed because read operations running with `repeatable read` isolation level, and `delete`, `insert`, and `update` operations use locks that are released only after the transaction terminates.

To change the `NumberOfLocks`, see Clearing and Archiving History Files

##### Example 3–18 Example lock information

For example:

    NodeNO Avail Free Waits
    0 50000 20000 101 50000 20000 0

### Log Buffer Information

Log buffer information is:

- `NodeNo`: Node Number

- `Available`: amount of memory allocated for the log buffer in MB

- `Free`: amount of free memory in MB

Do not worry if free space is very small, since HADB starts compressing the log buffer. HADB starts compression from the head of the ring buffer and performs it on consecutive log records. Compression cannot proceed when HADB encounters a log record that has not been executed by the node and received by the mirror node

##### Example 3–19 Example of log buffer information

For example:

    NodeNO Avail Free
    0 16 21 16 3

### Node Internal Log Buffer Information

Node internal log buffer information is:

- Node Number

- Available: amount of memory allocated for the log device in MB

- Free: amount of free memory in MB

##### Example 3–20 Example of internal log buffer information

For example:

`NodeNO Avail Free`

`0 16 21 16 3`

# Maintaining HADB Machines

HADB achieves fault tolerance by replicating data on mirror nodes. In a production environment, a mirror node is on a separate DRU from the node it mirrors, as described in Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Deployment Planning Guide.

A failure is an unexpected event such as a hardware failure, power failure, or operating system reboot. The HADB tolerates single failures: of one node, one machine (that has no mirror node pairs), one or more machines belonging to the same DRU, or even one entire DRU. However, HADB does not automatically recover from a double failure, which is the simultaneous failure of one or more mirror node pairs. If a double failure occurs, you must clear HADB and recreate its session store, which erases all its data.

There are different maintenance procedures, depending on whether you need to work on a single machine or multiple machines.

## To perform maintenance on a single machine

This procedure is applicable to both planned and unplanned maintenance, and does not interrupt HADB availability.

1. Perform the maintenance procedure and get the machine up and running.

2. Ensure that `ma` is running.

   If `ma` runs as a Windows service or under `init.d` scripts (recommended for deployment), it should have been started by the operating system. If not start it manually. See Starting the Management Agent.

3. Start all nodes on the machine.

   For more information, see Starting a Node.

4. Check whether the nodes are active and running.

   For more information, see Getting the Status of HADB

## To perform planned maintenance on all HADB machines

Planned maintenance includes operations such as hardware and software upgrades. This procedure does not interrupt HADB availability.

1. For each spare machine in the first DRU, repeat the single machine procedure as described in To perform maintenance on a single machine, one by one, for each machine.

2. For each active machine in the first DRU, repeat the single machine procedure as described in To perform maintenance on a single machine, one by one, for each machine.

3. Repeat step 1 and step 2 for the second DRU.

## To perform planned maintenance on all HADB machines

This procedure is applicable when HADB is on single or multiple machines. It interrupts HADB service during the maintenance procedure.

1. Stop HADB. See Stopping a Database .

2. Perform the maintenance procedure and get all the machines up and running.

3. Ensure `ma` is running.

4. Start HADB.

   For more information, see Starting a Database.

   After you complete the last step, HADB data becomes available again.

## To perform unplanned maintenance in the event of a failure

1. Check the database state.

   See Getting the Status of HADB

   - If the database state is Operational or better:

     The machines needing unplanned maintenance do not include mirror nodes. Follow the single machine procedure for each failed machine, one DRU at a time. HADB service is not interrupted.

   - If the database state is Non-Operational:

     The machines needing unplanned maintenance include mirror nodes. One such case is when the entire HADB is on a single failed machine. Get all the machines up and running first. Then clear HADB and recreate the session store. See Clearing a database. This interrupts HADB service.

## Clearing and Archiving History Files

HADB history files record all database operations and error messages. HADB appends to the end of existing history files, so the files grow over time. To save disk space and prevent files from getting too large, periodically clear and archive history files.

To clear a database’s history files, use the `hadbm clearhistory` command.

The command syntax is:

    hadbm clearhistory
    [--saveto=path]
    [dbname]
    [--adminpassword=password | --adminpasswordfile=file]
    [--agent=maurl]

The dbname operand specifies the database name. The default is `hadb`.

Use the `--saveto` option (short form `-o`) to specify the directory in which to store the old history files. This directory must have appropriate write permissions. See General Options for a description of other command options.

For more information, see hadbm-clearhistory(1).

The `--historypath` option of the `hadbm create` command determines the location of the history files. The names of the history files are of the format dbname`.out.` nodeno. For information on `hadbm create`, see Creating a Database

### History File Format

Each message in the history file contains the following information:

- The abbreviated name of the HADB process that produced the message.

- The type of message:

  - `INF` - general information

  - `WRN` - warnings

  - `ERR` - errors

  - `DBG` - debug information

- A timestamp. The time is obtained from the host machine system clock.

- The service set changes occurring in the system when a node stops or starts.

Messages about resource shortages contain the string “HIGH LOAD.”

You do not need a detailed knowledge of all entries in the history file. If for any reason you need to study a history file in greater detail, contact Sun customer support.

- Previous: Chapter 2 Installing and Setting Up High Availability Database
- Next: Chapter 4 Configuring Load Balancing and Failover

- © 2010, Oracle Corporation and/or its affiliates