C H A P T E R 8

Recovering From Catastrophic Failure

Certain events can be classified as catastrophic failures, such as flooding in a computer room. This chapter describes the procedure to follow after such an event. You might require the assistance of your ASP or Sun Microsystems customer support.

This chapter contains the following sections:

- Recovery Task Overview

- Recovery Procedures

##

Recovery Task Overview

You should not recover any system component, software element, or SAM-QFS file system that has not failed. However, you might need to reconfigure the SAM-QFS file system on a restored system to regain access to file systems or to determine whether any file system has failed. For details in performing these tasks, see the other sections of this chapter.

The process of recovering from a catastrophic failure involves the following steps:

1. Determining the failed system component

See To Restore Failed System Components.

2. Disabling the archiver and the recycler until all files are restored

See To Disable the Archiver and Recycler Until All Files Are Restored.

3. Comparing previous and current configuration files, and reconciling inconsistencies

See To Keep and Compare Previous and Current Configuration and Log Files.

4. Repairing disks

See To Repair Disks.

5. Restoring or building new library catalog files

See To Restore or Build New Library Catalog Files.

6. Making new file systems and restoring from `samfsdump` output

See To Make New File Systems and Restore From samfsdump Output.

##

Recovery Procedures

This section details the procedures involved in recovering from a catastrophic failure.

### To Restore Failed System Components

1. Ascertain which components have failed.

2. If a hardware component has failed, restore it to operation, preserving any available data.

If the failing component is a disk drive that has not totally failed, preserve as much information as possible. Before replacing or reformatting the disk, identify any salvageable files, and copy these files to a tape or to another disk for future use in the recovery process. Salvageable files to identify and copy include the following:

- SAM-QFS file system dumps

- Sun StorEdge SAM-FS configuration files, archiver log files, or library catalogs

3. If the Solaris Operating System (OS) has failed, restore it to operation.

See Recovering From Failure of the Operating Environment Disk. Verify that the Solaris OS is functioning correctly before proceeding.

4. If the Sun StorEdge SAM-FS or Sun StorEdge QFS package has been damaged, remove and reinstall it from a backup copy or from its distribution file.

You can verify whether a package has been damaged by using the `pkgchk`(1M) utility.

5. If disk hardware used by the Sun StorEdge SAM-FS software was repaired or replaced in Step 2, configure the disks (for RAID binding or mirroring) as necessary.

Reformat disks only if they have been replaced or if it is otherwise absolutely necessary.

### To Disable the Archiver and Recycler Until All Files Are Restored

| Caution - If the recycler is enabled so that it runs before all files are restored, cartridges with good archive copies might be improperly relabeled.

1. Add a single global `wait` directive to the `archiver.cmd` file or add a file-system-specific wait directive for each file system for which you want to disable archiving.

a. Open the `/etc/opt/SUNWsamfs/archiver.cmd` file for editing and find the section in which you want to insert the `wait` directive.

In the following sample file, local archiving directives exist for two file systems, `samfs1` and `samfs2`.

    # vi /etc/opt/SUNWsamfs/archiver.cmd
    ...
    fs = samfs1
    allfiles   .
    1   10s
    fs = samfs2
    allfiles   .
    1   10s

b. Add the `wait` directive.

- To apply the directive globally, insert it before the first `fs =` command (`fs = samfs1`), as shown here:

    wait
    fs = samfs1
    allfiles   .
    1   10s
    fs = samfs2
    allfiles   .
    1   10s
    :wq

- To apply the directive to a single file system, insert it after the `fs =` command for that file system, as shown here:

    fs = samfs1
    wait
    allfiles   .
    1   10s
    fs = samfs2
    wait
    allfiles   .
    1   10s
    :wq

2. Add a global `ignore` directive to the `recycler.cmd` file, or add a file-system-specific `ignore` directive for each library for which you want to disable recycling.

a. Open the `/etc/opt/SUNWsamfs/recycler.cmd` file for editing, as shown in the following example.

    # vi /etc/opt/SUNWsamfs/recycler.cmd
    ...
             logfile = /var/adm/recycler.log
             lt20 -hwm 75 -mingain 60
             lt20 75 60
             hp30 -hwm 90 -mingain 60 -mail root
             gr47 -hwm 95 -mingain 60 -mail root

b. Add the `ignore` directives.

The following example shows `ignore` directives added for three libraries.

    #  recycler.cmd.after  - example recycler.cmd file
    #
             logfile = /var/adm/recycler.log
             lt20 -hwm 75 -mingain 60 -ignore
             hp30 -hwm 90 -mingain 60 -ignore -mail root
             gr47 -hwm 95 -mingain 60 -ignore -mail root

### To Keep and Compare Previous and Current Configuration and Log Files

Follow these steps before rebuilding the system.

1. Recover any available Sun StorEdge SAM-FS configuration files or archiver log files from the system's disks.

2. Compare the restored versions of all configuration files represented in the ```SAMreport` with those restored from the system backups.

3. If inconsistencies exist, determine the effect of the inconsistencies and reinstall the Sun StorEdge QFS file system, if necessary, using the configuration information in the `SAMreport` file.

For more information on `SAMreport` file, see the `samexplorer`(1M) man page.

### To Repair Disks

For SAM-QFS file systems that reside on disks that have not been replaced, run the `samfsck`(1M) utility to repair small inconsistencies, reclaim lost blocks, and so on.

For command-line options to the `samfsck` utility, see the `samfsck`(1M) man page.

### To Restore or Build New Library Catalog Files

1. Replace the most recent library catalog file copies from the removable media files, from the Sun StorEdge SAM-FS server disks, or from the most recent file system archive copies.

2. If the library catalogs are unavailable, build new catalogs by using the `build.cat`(1M) command and the library catalog section of the most recent `SAMreport` as input.

Use the newest library catalog copy available for each automated library.

Note - Sun StorEdge SAM-FS systems automatically rebuild library catalogs for SCSI-attached automated libraries. This does not occur for ACSLS-attached automated libraries. Tape usage statistics are lost.

### To Make New File Systems and Restore From samfsdump Output

Follow these steps for SAM-QFS file systems that were partially or completely resident on disks that were replaced or reformatted.

1. Obtain the most recent copy of the `samfsdump`(1M) output file.

2. Make a new file system and restore the SAM-QFS file system using the `samfsdump` output file.

a. Use the `sammkfs`(1M) command to make a new file system.

    # mkdir /sam1
    # sammkfs samfs1
    # mount samfs1

b. Use the `samfsrestore`(1M) command with the `-f` option and the `-g` option, use the following syntax:

    samfsrestore -f output-file-location -g log-file

where:

- output-file-location is the location of the `samfsdump` output file.

- log-file is the path name of the new log file that will list all the files that were online.

For example:

    # cd /sam1
    # samfsrestore -f /dump_sam1/dumps/040120 -g /var/adm/messages/restore_log

Note - Once all file systems have been restored, the system can be made available to users in degraded mode.

3. On the file systems you have just restored, perform the following steps:

a. Run the `restore.sh`(1M) script against the log file, and stage all files that were known to be online before the outage. In a shared environment, this script must be run on the metadata server.

b. Run the `sfind`(1M) command against the SAM-QFS file system to determine which files are labeled as damaged.

These files might or might not be restorable from tape, depending on the content of the archive log files. Determine the most recently available archive log files from one of the following sources, in this order:

- The removable media file.

- The Sun StorEdge SAM server disk.

- The most recent file system archive. This source is likely to be slightly outdated.

c. Run the `grep`(1) command against the most recent archive log file to search for the damaged files.

This will enable you to determine whether any of the damaged files were archived to tape after the last time the `samfsdump`(1M) command was run.

d. Examine the archive log files to identify any archived files that do not exist in the file system.

e. Use the `star`(1M) command to restore the damaged and nonexistent files identified in Step c and Step d.

4. Reimplement disaster recovery scripts, methods, and `cron`(1M) jobs using information from the backup copies.

Copyright © 2006, Sun Microsystems, Inc. All Rights Reserved.