C H A P T E R 6

Staging

Staging is the process of copying file data from nearline or offline storage back to online storage. The staging capabilities enable you to stage files immediately, to never stage files, to specify partial staging, and to specify other staging actions. The never-stage capability can be used, for example, by applications that randomly access small records from large files; when this is enabled, the data is accessed directly from the archive media without staging the file online.

This chapter describes the Sun StorEdge SAM-FS file staging capability. It contains the following topics:

- The stager.cmd File

- The archiver.cmd File's Role in Staging

- Prioritizing Preview Requests Using the preview.cmd File

- Calculating Total Preview Request Priority

- How to Set Up a Preview Request Priority Scheme

##

The `stager.cmd` File

You can use the `stager.cmd` file to specify the stager's behavior. The full path name to this file is `/etc/opt/SUNWsamfs/stager.cmd`. By default, the stager performs the following actions:

- The stager attempts to use all the drives in the library to stage files.

- The stage buffer size is determined by the media type, and the stage buffer is not locked.

- No log file is written.

- Up to 1000 stage requests can be active at any one time.

The `stager.cmd` file allows you to specify directives to override the default behaviors. The rest of this section describes the stager directives. For additional information on stager directives, see the `stager.cmd`(4) man page.

The Example stager.cmd File shows the completed `stager.cmd` file after all possible directives have been set.

CODE EXAMPLE 6-1 shows the example `mcf` file used by the examples in this chapter.

CODE EXAMPLE 6-1 `mcf` File Used in this Chapter's Examples
    #
    # Sun StorEdge SAM-FS file system configuration example
    #
    # Equipment       Eq Eq Family Dev Additional
    # Identifier      Or Tp Set    St  Parameters
    # --------------- -- -- ------ --- ----------
    samfs1            60 ms samfs1
    /dev/dsk/c1t1d0s6 61 md samfs1 on
    /dev/dsk/c2t1d0s6 62 md samfs1 on
    /dev/dsk/c3t1d0s6 63 md samfs1 on
    /dev/dsk/c4t1d0s6 64 md samfs1 on
    /dev/dsk/c5t1d0s6 65 md samfs1 on
    #
    samfs2             2 ms samfs2
    /dev/dsk/c1t1d0s0 15 md samfs2 on
    /dev/dsk/c1t0d0s1 16 md samfs2 on
    #
    /dev/samst/c0t2d0 20 od -      on
    /dev/samst/c1t2u0 30 rb dog  on /var/opt/SUNWsamfs/catalog/dogcat
    /dev/samst/c1t5u0 31 od dog    on
    /dev/samst/c1t6u0 32 od dog    on
    /dev/rmt/0cbn     40 od -      on
    /dev/samst/c1t3u1 50 rb bird on /var/opt/SUNWsamfs/catalog/birdcat
    /dev/rmt/2cbn     51 tp bird   on

### To Create or Modify a `stager.cmd` File and Propagate Your Changes

1. Use `vi`(1) or another editor to edit the `stager.cmd` file.

The full path to this file is as follows:

    /etc/opt/SUNWsamfs/stager.cmd

For information on the directives you can include in this file, see the following subsections:

- Specifying the Number of Drives

- Setting the Stage Buffer Size

- Specifying a Log File

- Specifying the Number of Stage Requests

2. Save and close the `stager.cmd` file.

3. Use the `samd`(1M) command with its `config` option to propagate the file changes and restart the system.

    # samd config

### Specifying the Number of Drives

By default, the stager uses all available drives when staging files. If the stager keeps all the drives busy, this can interfere with the archiver's activities. The `drives` directive specifies the number of drives available to the stager. This directive has the following format:

    drives = library count

TABLE 6-1 Arguments for the drives Directive
Argument | Meaning
--- | ---
library | The Family Set name of a library as it appears in the Sun StorEdge SAM-FS `mcf` file.
count | The maximum number of drives to be used. By default, this is the number of drives configured in the `mcf` file for this library.

For example, the following directive line specifies that only one drive from the `dog` family set's library be used for staging files:

    drives = dog 1

For more information on the `mcf` file, see the `mcf`(4) man page.

### Setting the Stage Buffer Size

By default, a file being staged is read into memory in a buffer prior to restoring the file from the archive media back to online disk cache. You can use the `bufsize` directive to specify a nondefault buffer size and, optionally, to lock the buffer. These actions can improve performance, and you can experiment with various buffer_size values. This directive has the following format:

    bufsize = media buffer_size [ lock ]

TABLE 6-2 Arguments for the bufsize Directive
Argument | Meaning
--- | ---
media | Specify the archive media type from the list on the mcf(4) man page.
buffer_size | Specify a number from 2 through 32. The default is 4. This value is multiplied by the dev`_blksize` value for the media type, and the resulting buffer size is used. The dev`_blksize` can be specified in the `defaults.conf` file. The higher the number specified for buffer_size, the more memory is used. For more information on this file, see the `defaults.conf`(4) man page.
`lock` | The `lock` argument indicates whether or not the archiver should use locked buffers when staging archive copies. If `lock` is specified, the stager sets file locks on the stage buffer in memory for the duration of the copy operation. This avoids the overhead of locking and unlocking the buffer for each I/O request and can result in a reduction in system CPU time. The `lock` argument should be specified only on large systems with large amounts of memory. Insufficient memory can cause an out-of-memory condition. The `lock` argument is effective only if direct I/O is enabled for the file being staged. By default, `lock` is not specified and the file system sets the locks on all direct I/O buffers, including those for staging. For more information on enabling direct I/O, see the `setfa`(1) man page, the `sam_setfa`(3) library routine man page, or the `-O forcedirectio` option on the `mount_samfs`(1M) man page.

For example, this directive can be specified in the `stager.cmd` file in a line such as the following:

    bufsize=od 8 lock

### Specifying a Log File

You can request that the Sun StorEdge SAM-FS file system collect file-staging event information and write it to a log file. The `logfile` directive specifies a log file to which the stager can write logging information. This directive has the following format:

    logfile=filename [ event ]

For filename, specify a full path name.

For event, specify one or more staging events. If you specify more than one event, use spaces to separate each event. Events enabled by default are as follows: `finish cancel error`. Possible event specifications are as follows:

TABLE 6-3 Keywords for the event Argument
event | Action
--- | ---
`all` | Logs all staging events.
`start` | Logs when staging begins for a file.
`finish` | Logs when staging ends for a file. Enabled by default.
`cancel` | Logs when the operator cancels a stage. Enabled by default.
`error` | Logs staging errors. Enabled by default.

When a log file is specified, the stager writes one or more lines to the log file for each file staged. This line includes information such as the name of the file, the date and time of the stage, and the VSN. For example, the following directive line specifies file `/var/adm/stage.log`:

    logfile=/var/adm/stage.log

CODE EXAMPLE 6-2 shows an example of a stager log file.

CODE EXAMPLE 6-2 Stager Log File Example
    S 2003/12/16 14:06:27 dk disk01 e.76d 2557.1759 1743132 /sam1/testdir0/filebu 1 root other root 0
    F 2003/12/16 14:06:27 dk disk01 e.76d 2557.1759 1743132 /sam1/testdir0/filebu 1 root other root 0
    S 2003/12/16 14:06:27 dk disk02 4.a68 1218.1387 519464 /sam1/testdir1/fileaq 1 root other root 0
    S 2003/12/16 14:06:43 dk disk01 13.ba5 3179.41 750880 /sam1/testdir0/filecl 1 root other root 0
    F 2003/12/16 14:06:43 dk disk01 13.ba5 3179.41 750880 /sam1/testdir0/filecl 1 root other root 0
    S 2003/12/16 14:06:59 dk disk01 17.167b 1155.1677 1354160 /sam1/testdir0/filedb 1 root other root 0
    F 2003/12/16 14:06:59 dk disk01 17.167b 1155.1677 1354160 /sam1/testdir0/filedb 1 root other root 0
    S 2003/12/16 14:06:59 dk disk02 f.f82 3501.115 1458848 /sam1/testdir1/filecb 1 root other root 0
    S 2003/12/16 14:07:15 dk disk01 1f.473 1368.1419 636473 /sam1/testdir0/fileed 1 root other root 0
    S 2003/12/16 14:07:15 dk disk02 16.f15 3362.45 1065457 /sam1/testdir1/filecz 1 root other root 0
    S 2003/12/16 14:07:31 dk disk01 23.201d 3005.1381 556807 /sam1/testdir0/fileeq 1 root other root 0
    S 2003/12/16 14:07:47 dk disk01 26.c4d 2831.1113 1428718 /sam1/testdir0/fileez 1 root other root 0
    S 2003/12/16 14:07:47 dk disk02 1b.835 3736.59 1787855 /sam1/testdir1/filedp 1 root other root 0

As TABLE 6-4 shows, the stager log file consists of lines of information divided into nine fields. TABLE 6-4 describes the content of the stager log file fields.

TABLE 6-4 Stager Log File Fields
Field | Content Description
--- | ---
1 | Stage activity. `S` for start. `C` for canceled. `E` for error. `F` for finished.
2 | Date of stage action in yyyy`/`mm`/`dd format.
3 | Time of stage action in hh`:`mm`:`ss format.
4 | Archive media type. For information on media types, see the `mcf`(4) man page.
5 | VSN.
6 | Physical position of start of archive file on media (`tar`(1) file) and file offset on the archive file in hexadecimal.
7 | Inode number and generation number. The generation number is an additional number used in addition ot the inode number for uniqueness since inode numbers get re-used.
8 | Length of file.
9 | Name of file.
10 | Archive copy number.
11 | The file's user ID.
12 | The file's group ID.
13 | The requestor's group ID.
14 | The Equipment Ordinal of the drive from which the file was staged.

### Specifying the Number of Stage Requests

You can specify the number of stage requests that can be active at any one time by using the `maxactive` directive. This directive has the following format:

    maxactive=number

By default, number is 4000. The minimum number allowed is 1.

For example, the following directive line specifies that no more than 500 stage requests can be in the queue simultaneously:

    maxactive=500

### Example `stager.cmd` File

CODE EXAMPLE 6-3 shows an example `stager.cmd` file.

CODE EXAMPLE 6-3 Example `stager.cmd` File
    # This is stager.cmd file /etc/opt/SUNWsamfs/stager.cmd
    drives=dog 1
    bufsize=od 8 lock
    logfile=/var/adm/stage.log
    maxactive=500

##

The `archiver.cmd` File's Role in Staging

Most directives in the `archiver.cmd` file affect archiving, but the archive set assignment directive allows you to specify stage attributes that apply to all files in an archive set. The archive set assignment directive has the following format:

    archive_set_name path [search_criteria ...] directives ... ]

The chapter called Archiving describes the archive set assignment directive and its arguments completely. TABLE 6-5 shows the staging directives that can appear as directives in an archive set assignment directive.

TABLE 6-5 Staging directives that can Appear in the `archiver.cmd` File
Directive | Effect
--- | ---
`-stage a` | Specifies that the files in the archive set should be associatively staged.
`-stage n` | Specifies that the files in the archive set should never be staged.

For more information on these and the other `archiver.cmd` directives, see Archiving.

##

Prioritizing Preview Requests Using the `preview.cmd` File

The archiver and stager processes both can request that media be loaded and unloaded. If the number of requests exceeds the number of drives available for media loads, the excess number of requests is sent to the preview queue.

Archive and stage requests in the preview queue are those that cannot be immediately satisfied. By default, preview requests are satisfied in first-in-first-out (FIFO) order.

The number of entries that can be in the preview queue is determined by the `previews=` directive in the `defaults.conf` file. For information on changing the value of this directive, see the `defaults.conf`(4) man page.

You can assign different priorities to preview requests. You can override the FIFO default by entering directives in the preview command file, which is written to the following location:

    /etc/opt/SUNWsamfs/preview.cmd

This file schedules preview requests based on whether the request is for file staging or archiving. You can also increase the priority for specific VSNs. Further, settings in the `preview.cmd` file can also reprioritize preview requests for all or for specific file systems based on the high watermark (HWM) or low watermark (LWM) settings.

The `sam-amld` daemon reads the preview directives at startup. You must specify the directives one per line. If you change this file while the `sam-amld` daemon is running, you have to restart the `sam-amld` daemon to have them take effect. Comment lines begin with a pound sign (#) and extend through the end of the line. For more information on this file, see the `preview.cmd`(4) man page.

The following two types of directives can appear in the `preview.cmd` file:

- Global directives, which apply to all file systems. These must appear before the first `fs =` line.

- Directive that are specific to a file system, which follow the global directives. Like the `archiver.cmd` file, the `preview.cmd` file can contain directives specific to individual file systems. The directives specific to individual file systems must appear in the file after all global directives.

The file system directives must begin with an `fs =` file_system_name directive. This directive names the file system to which all subsequent directives pertain. More than one block of file directives can appear in a file. File system directives apply until the next `fs =` line is encountered or until the end of file is encountered.

Note - When multiple directives affect a file system, the directives that are specific to a particular file system override the global directives.

### VSN and Age Directives (Global)

The VSN and age priority directives are global directives. If they are present in your `preview.cmd` file, they must appear before any directives that are specific to a file system. That is, they must appear prior to any `fs =` directives. The VSN priority directive has the following format:

    vsn_priority = value

This directive is a static priority factor. It indicates the value by which the total priority increases for a VSN flagged as a high-priority VSN. The default value for `vsn_priority` is 1000.0. VSNs must have their priority flag set when they are scheduled as preview requests to gain this value. Use the `chmed`(1M) command to set the priority flag with the `p` option (for example, `chmed +p lt.AAA123`). Setting this flag takes effect for all submitted requests for the VSN that are not already preview requests. The age priority directive has the following format:

    age_priority = factor

This directive is a static priority factor. Its overall effect is dynamic. The `age_priority` factor is multiplied by the number of seconds a request is a preview request. The result is added to the overall priority of the request. The longer a request waits to be satisfied, the larger the age factor becomes. Setting this factor helps to ensure that older requests are not indefinitely superseded by newer requests with other higher-priority factors.

If this factor is more than 1.0, it increases the importance of the time factor in calculating the total priority. If it is less than 1.0, it decreases the importance of the time factor. Setting the factor to 0.0 eliminates the time factor from the overall priority calculation.

A VSN whose priority flag is not set increases in priority based on the time it remains in the queue. Its priority can become higher than a VSN that comes into the queue later with the priority flag already set.

### Watermark Directives (Global or File System Specific)

The watermark preview request directives can be used as either global or file system specific directives. The watermark priority directives determine the watermark priority (`wm_priority`) of the preview requests. CODE EXAMPLE 6-4 shows that the `wm_priority` factor is the sum of several settings.

CODE EXAMPLE 6-4 `wm_priority` Calculation
    lwm_priority +
    lhwm_priority +
    hlwm_priority +
    hwm_priority
    __________________
    = wm_priority

When the `wm_priority` factor is a positive number, the result on the overall calculated priorities increases archiving requests over staging requests. However, the `wm_priority` factor can also be a negative number. In this case, the overall priority for archiving requests is reduced, which tends to favor staging requests over archival requests. A setting of 0.0 (or no specified command at all) indicates that no special action occurs to archival requests when the file system is in this condition. For more information on this, see the example in Example 1: Enforcing Stage Requests.

TABLE 6-6 shows the four watermark priority directives and their arguments

TABLE 6-6 Watermark Priority Directives
Priority Directive | Argument
--- | ---
`lwm_priority =` value | For value, specify the amount by which the `wm_priority` factor changes for archiving requests when the file system is below the LWM level. The default is 0.0.
`lhwm_priority =` value | For value, specify the amount by which the `wm_priority` factor changes for archiving requests when the file system crosses from below to above the LWM but remains below the HWM level. This generally indicates that the file system is filling up. The default is 0.0.
`hlwm_priority =` value | For value, specify the amount by which the `wm_priority` factor changes for archiving requests when the file system has crossed from above to below the HWM but remains above the LWM level. This generally indicates that the releaser was not able to free enough disk space to leave the file system below LWM. The default is 0.0.
`hwm_priority =` value | For value, specify the amount by which the `wm_priority` factor changes for archiving requests when the file system is above the HWM level. The default is 0.0.

Together, the four watermark settings create a dynamic priority factor that includes a percentage value indicating how full the file system is and the levels at which the HWM and LWM are set. The value assigned to a preview request is determined by whether a factor is global, specific to a file system, or not set.

When a file system crosses from one condition to another, the priority of each VSN associated with that file system is recalculated based on the appropriate watermark priority setting, with or without the `chmed`(1M) command's `p` option.

The watermark priorities are used only to calculate media requests for archiving. They are not used to calculate media requests for staging.

The following example directives show how to slightly increase the priority for archiving requests when the file system is at HLWM. CODE EXAMPLE 6-5 shows the settings to use to enable the releaser to free enough disk space so that the file system gets below LWM.

CODE EXAMPLE 6-5 Settings for Going Below the LWM
    lhwm_priority = -200.0
    hlwm_priority = 100.0

##

Calculating Total Preview Request Priority

The numeric priority of preview requests is determined by combining several static and dynamic factors. Higher numbers correspond to higher priority. A static priority factor is set when the request is generated. Its effect does not change the overall priority after the request is generated and is waiting to be satisfied. A dynamic priority factor can increase or decrease the overall priority of a request while the request is waiting to be satisfied.

The total priority for a preview request is the sum of all priority factors. It is calculated as follows:

    total priority = vsn_priority + wm_priority + (age_priority * time_in_sec_as_preview_request)

##

How to Set Up a Preview Request Priority Scheme

It is necessary to change the default preview request FIFO scheme only when there are compelling system reasons to do so. The following possible conditions might necessitate changing the default preview request FIFO scheme:

- Condition 1: Ensure that staging requests are processed before archive requests.

- Condition 2: Ensure that archive requests gain top priority when a file system is about to fill up.

- Condition 3: Push requests that use a specific group of media to the top of the preview request list.

For environments in which user access to data is of paramount importance, the VSN drives are limited, or file archival is performed as a background function, you can use the `preview.cmd` file to influence how the storage system resources service the staging requests. You can customize the settings in the `preview.cmd` file to support any of the preceding scenarios and influence the configured Sun StorEdge SAM-FS environment.

Because data is not affected by the settings in this file, you are encouraged to experiment and adjust the directive settings to achieve the proper balance between archiving and staging requests when weighed against the priorities of each preview request.

CODE EXAMPLE 6-6 shows an example `preview.cmd` file that addresses the three conditions listed previously.

CODE EXAMPLE 6-6 Example `preview.cmd` File
    # condition 1
    lwm_priority = -200.0
    lhwm_priority = -200.0
    hlwm_priority = -200.0
    # condition 2
    hwm_priority = 500.0
    # condition 3
    age_priority = 1.0

### Example 1: Enforcing Stage Requests

The following example settings demonstrate one way to ensure that stage requests have priority over archive requests. This example assumes the following:

- Several requests are sitting in the queue for 100 seconds.

- The default `vsn_priority` is 1000.

TABLE 6-7 shows how the total request priorities are calculated.

TABLE 6-7 Request Priority Example
Priority | Calculation
--- | ---
Archive VSN with priority, LWM: | 1000 + (-200) + (1 x 100) = 900
Stage VSN with priority, LWM: | 1000 + 0 + (1 x 100) = 1100
Stage VSN without priority, LWM: | 0 + 0 + (1 x 100) = 100

This example shows that a negative value for `wm_priority` tends to favor staging requests over archival requests when the other factors are equal.

### Example 2: Enforcing Archive Requests

When the environment is balanced between the importance of staging a file back to the user versus getting new files archived to media, the biggest concern is exceeding the HWM. In this situation, if there are not enough files who have met their archive requirements to lower the percent full of the file system, completing the pending archive requests is the next best way to keep the file system from filling up.

In this situation, the `preview.cmd` file can be as simple as the following:

    hwm_priority = 500.0

### Example 3: Prioritizing Requests by Media

In project-oriented environments, specific users might be working on groups of files that use specific VSNs and are segregated from other users. In this environment, certain projects might have higher priorities at certain times; hence, greater priority might be required from the available system storage resources. You can configure the `preview.cmd` file with the following directive to give users and their media the appropriate priority for media drives:

    hwm_priority = 5000.0

Then, for every VSN in the priority user's group, enter the following information:

    # chmed +p lt.AAA123 ## or whatever VSN is used

Thereafter, every request that requires VSN `AAA123` (or whatever VSN is used) is placed above other pending mount requests in the preview queue.

Later, to deprioritize the user's media, enter the following reverse command for every VSN:

    # chmed -p lt.AAA123 ## or whatever media type is used

### Example 4: Complex Prioritization

Assume that there are two Sun StorEdge SAM-FS file systems with the following requirements:

- No request should sit too long in the queue (`age_priority`).

- When a file system is below the LWM, staging requests should take precedence.

- When a file system is above the LWM but below the HWM, it is not necessary to prioritize archive or stage requests one over the other. CODE EXAMPLE 6-7 shows the affected directives.

CODE EXAMPLE 6-7 Directives
    lwm_priority =  -200.0
    lhwm_priority = 0.0
    hlwm_priority = 0.0

In this case, the other directives remain unchanged.

When a file system goes over the HWM, archive requests should take priority.

If both file systems are over the HWM, it is more important to prevent the second file system (for example, `samfs2`) from filling up. This might occur if, for example, `samfs1` is a user working file system and `samfs2` is the critical-system file system.

In all cases, regardless of the situation, a request for a select group of VSNs takes precedence in the preview request queue if the `chmed`(1M) command's `p` flag is set.

CODE EXAMPLE 6-8 shows a `preview.cmd` file that prioritizes requests according to the requirements in the preceding list.

CODE EXAMPLE 6-8 The `preview.cmd` File
    age_priority = 100.0
    vsn_priority = 20000.0
    lhwm_priority = -200.0
    hlwm_priority = -200.0
    fs = samfs1
    hwm_priority = 1000.0
    fs = samfs2
    hwm_priority = 5000.0

Copyright © 2004, Sun Microsystems, Inc. All rights reserved.