Oracle® Database

4.10.1 Overview of Device Naming Persistence Changes

With the Linux 2.6 kernel, support for raw devices is deprecated. The preferred storage access is direct input/output to block devices using O_DIRECT. As a result of this change, the RHEL4 and OEL4 file /etc/sysconfig/rawdevice and the RHEL5 and OEL5 file /etc/udev/rules.d/60-raw.rules are deprecated. For details, refer to the Linux documentation for your 2.6 kernel.

The 2.4 kernel device file naming scheme devlabel maintained persistent device file names between server restarts. By default, the 2.6 kernel device file naming scheme udev dynamically creates device file names when the server is started, and assigns ownership of them to root. If udev applies default settings, then it changes device file names and owners for voting disks or Oracle Cluster Registry partitions, corrupting them when the server is restarted. For example, a voting disk on a device named /dev/sdd owned by the user crs may be on a device named /dev/sdf owned by root after restarting the server.

To prevent corruption, you need to create a custom rules file. When udev is started, it sequentially carries out rules (configuration directives) defined in rule files. These files are in the path /etc/udev/rules.d/. Rules files are read in lexical order. For example, rules in file 10-wacom.rules are parsed and carried out before rules in rule file 90-ib.rules. Where rules files describe the same devices, on Asianux, Red Hat, and Oracle Enterprise Linux, the last file read is the one that is applied. (On SUSE 2.6 kernels, it is the first file read).

4.10.2 Configuring Raw Devices for Clusterware on Red Hat and Oracle Enterprise Linux 5

This section contains the following topics:

• Configure SCSI_ID to Return Unique Device Identifiers

• Configure Udev for Persistent Naming of Oracle Clusterware Devices

• Bind Raw Devices Using Udev

• Verify Persistent Oracle Clusterware Storage Devices

Configure SCSI_ID to Return Unique Device Identifiers

Before you can configure udev to name devices, you must first configure scsi_id to return device identifiers, and then ensure that these devices are visible and accessible on all cluster nodes. To do this, complete the following task:

1. Modify the /etc/scsi_id.config file by adding or replacing the 'option=-b' parameter/value pair (if it exists) with 'option=-g'. For example:

   # cd /etc
   # cp scsi_id.config scsi_id.config.orig
   # grep -v ^# /etc/scsi_id.config
   vendor="ATA",options=-p 0x80
   options=-g
   

2. Run the command fdisk (/sbin/fdisk) to ensure that Clusterware devices are visible. For example:

   # /sbin/fdisk -l /dev/sdb1 /dev/sde1
    
   Disk /dev/sdb1: 261 MB, 261890048 bytes
   9 heads, 56 sectors/track, 1014 cylinders
   Units = cylinders of 504 * 512 = 258048 bytes
    
   Disk /dev/sdb1 doesn't contain a valid partition table
    
   Disk /dev/sde1: 52 MB, 52403200 bytes
   2 heads, 50 sectors/track, 1023 cylinders
   Units = cylinders of 100 * 512 = 51200 bytes
    
   Disk /dev/sde1 doesn't contain a valid partition table
   

   In some cases, to see newly provisioned or modified) devices on shared storage, you may need to update cluster node operating systems. Do this either by restarting the nodes, or by using commands such as /sbin/partprobe device, or sfdisk -r device. Resolve any issues preventing cluster nodes from correctly seeing or accessing storage devices you intend to use for Clusterware files before proceeding.

   Note:

   At this point, cluster nodes may refer to the devices using different device file names. This is expected.

3. Run the command scsi_id (/sbin/scsi_id) on storage devices from one cluster node to obtain their unique device identifiers. When running the scsi_id command with the -s argument, the device path and name passed should be that relative to the sysfs directory /sys (for example, /block/device) when referring to /sys/block/device. For example:

   # /sbin/scsi_id -g -s /block/sdb/sdb1
   360a98000686f6959684a453333524174
    
   # /sbin/scsi_id -g -s /block/sde/sde1
   360a98000686f6959684a453333524179
   

   Record the unique SCSI identifiers of Clusterware devices, so you can provide them when required in the following section, Configure Udev for Persistent Naming of Oracle Clusterware Devices.

   Note:

   The command scsi_id should return the same device identifier value for a given device, regardless of which node the command is run from.

Configure Udev for Persistent Naming of Oracle Clusterware Devices

Configure persistent user-defined naming of Oracle Clusterware device file names in a udev rules file. This step is optional, but recommended.

The default rule files affecting storage devices are rule files 50 and 51. So create a custom rules file using the format [number]-[name][.rules] with a number value greater than 51 to ensure that the device settings you provide are the ones applied. For example:

55-oracle-naming.rules

To do this, complete the following tasks:

1. Create a custom udev device naming rule file. For example:

   # touch /etc/udev/rules.d/55-oracle-naming.rules
   

2. Using the a text editor such as vi, add to the custom device naming rule file the device-matching rules for the storage devices you intend to use with Oracle Clusterware, matching them to the unique SCSI identifiers you determined in the preceding section. For example:

   # Configure persistent, user-defined Oracle Clusterware device file names
   KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", 
   RESULT=="360a98000686f6959684a453333524174", NAME="ocr1", OWNER="root", 
   GROUP="oinstall", MODE="0640"
   KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a453333524179", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"
   

   For each rule, if all specified keys (KERNEL, BUS, PROGRAM, RESULT) are matched, then the rule is applied and the specified assignments (NAME, OWNER, GROUP, MODE) are assigned to the device file name. However, if one or more keys are unmatched, then the rule is completely ignored and the default (arbitrary) kernel-assigned device file names are assigned to devices.

   Note:

   In the example rules files shown, Oracle Clusterware devices are created with oraInventory group (oinstall). Oracle recommends that you do this to ensure that you can run Cluster Verification Utility during installation.

3. Run the command udevtest (/sbin/udevtest) to test the udev rules configuration you have created. The output should indicate that the block devices are available and the rules are applied as expected. For example:

   # udevtest /block/sdb/sdb1
   main: looking at device '/block/sdb/sdb1' from subsystem 'block'
   udev_rules_get_name: add symlink
   'disk/by-id/scsi-360a98000686f6959684a453333524174-part1'
   udev_rules_get_name: add symlink
   'disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.887085-part1'
   udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has correct dev_t
   run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
   run_program: '/lib/udev/vol_id' returned with status 4
   run_program: '/sbin/scsi_id'
   run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a453333524174'
   run_program: '/sbin/scsi_id' returned with status 0
   udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1'
   udev_device_event: device '/block/sdb/sdb1' validate currently present symlinks
   udev_node_add: creating device node '/dev/ocr1', major = '8', minor = '17', 
   mode = '0640', uid = '0', gid = '500'
   udev_node_add: creating symlink
   '/dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1' to '../../ocr1'
   udev_node_add: creating symlink
   '/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085
   -part1' to '../../ocr1'
   main: run: 'socket:/org/kernel/udev/monitor'
   main: run: '/lib/udev/udev_run_devd'
   main: run: 'socket:/org/freedesktop/hal/udev_event'
   main: run: '/sbin/pam_console_apply /dev/ocr1
   /dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1
   /dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-
   part1'
   

   In the example output, note that applying the rules renames OCR device /dev/sdb1 to /dev/ocr1.

4. Restart the udev service by running the command start_udev (/sbin/start_udev). Restarting udev applies the udev rules to the devices, including the device file rules you have created. Use the command ls -l command to ensure that the rules file has applied the new device names the rules file has applied. For example:

   # start_udev
    
   # ls -l /dev | grep -e 'ocr1\|vote1'
   brw-r----- 1 root   oinstall 8, 17 Oct 29 15:31 ocr1
   brw-rw---- 1 oracle oinstall 8, 65 Oct 29 15:31 vote1
   

Bind Raw Devices Using Udev

1. If the file /etc/udev/rules.d/60-raw.rules does not exist, then create it. If it does exist, then create a rules file for raw devices used with Oracle installations. For example:

   # touch /etc/udev/rules.d/60-raw.rules
   

   or

   # touch /etc/udev/rules.d/61-oracleraw.rules
   

2. Add the udev raw binding rules to the raw devices rules file you created. For example:

   vi /etc/udev/rules.d/61-oracleraw.rules
   # Raw bind to Oracle Clusterware devices
   ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id",
    RESULT=="360a98000686f6959684a453333524174", RUN+="/bin/raw /dev/raw/raw1 %N"
   ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id",
    RESULT=="360a98000686f6959684a453333524179", RUN+="/bin/raw /dev/raw/raw2 %N"
    t 29 15:31 vote1
   

3. Create a udev raw permissions file /etc/udev/rules.d/65-raw-permissions.rules. For example:

   # touch /etc/udev/rules.d/65-raw-permissions.rules
   

4. Using a text editor, add the udev raw permission rules to the file /etc/udev/rules.d/65-raw-permissions.rules. For example:

   # Set permissions of raw bindings to Oracle Clusterware devices
   KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
   KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"
   

5. Test the udev rules by running the udevtest command (/sbin/udevtest) again to ensure that the rules are applied, and that they create proper permissions for Oracle Clusterware devices. For example:

   # udevtest /block/sdb/sdb1
   main: looking at device '/block/sdb/sdb1' from subsystem 'block'
   udev_rules_get_name: add symlink 'disk/by-id/scsi-360a98000686f69
   59684a453333524174-part1'
   udev_rules_get_name: add symlink 'disk/by-path/ip-192.168.1.1:3260
   -iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'
   udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has 
   correct dev_t
   run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
   run_program: '/lib/udev/vol_id' returned with status 4
   run_program: '/sbin/scsi_id'
   run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a45333
   3524174'
   run_program: '/sbin/scsi_id' returned with status 0
   udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1'
   udev_device_event: device '/block/sdb/sdb1' validate currently 
   present symlinks
   udev_node_add: creating device node '/dev/ocr1', major = '8', 
   minor = '17', mode = '0640', uid = '0', gid = '500'
   udev_node_add: creating symlink '/dev/disk/by-id/scsi-360a9800068
   6f6959684a453333524174-part1' to '../../ocr1'
   udev_node_add: creating symlink '/dev/disk/by-path/ip-192.168.1.1
   :3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1' to '../../ocr1'
   main: run: 'socket:/org/kernel/udev/monitor'
   main: run: '/lib/udev/udev_run_devd'
   main: run: 'socket:/org/freedesktop/hal/udev_event'
   main: run: '/sbin/pam_console_apply /dev/ocr1 /dev/disk/by-id/scsi-36
   0a98000686f6959684a453333524174-part1 /dev/disk/by-path/ip-192.168.1.
   1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'
   main: run: '/bin/raw /dev/raw/raw1 /dev/.tmp-8-17'
   

6. Restart udev to implement the rules you have created and tested. For example:

   # start_udev
   

Verify Persistent Oracle Clusterware Storage Devices

1. Use the fdisk command to check device naming. For example:

   # fdisk -l /dev/ocr1 /dev/vote1
    
   Disk /dev/ocr1: 261 MB, 261890048 bytes
   9 heads, 56 sectors/track, 1014 cylinders
   Units = cylinders of 504 * 512 = 258048 bytes
    
   Disk /dev/ocr1 doesn't contain a valid partition table
    
   Disk /dev/vote1: 52 MB, 52403200 bytes
   2 heads, 50 sectors/track, 1023 cylinders
   Units = cylinders of 100 * 512 = 51200 bytes
    
   Disk /dev/vote1 doesn't contain a valid partition table
   

2. Use the ls command to check device ownership. For example:

   # ls -l /dev | grep -ie 'ocr\|vote'
   brw-r----- 1   root dba  8, 17 Oct 29 15:31 ocr1
   brw-rw---- 1 oracle dba  8, 65 Oct 29 15:31 vote1
   

3. Use the udevinfo command to confirm unique SCSI device identifier mappings. For example:

   # udevinfo -q all -n /dev/ocr1
   P: /block/sdb/sdb1
   N: ocr1
   S: disk/by-id/scsi-360a98000686f6959684a453333524174-part1
   S: disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.87085-part1
   E: ID_VENDOR=NETAPP
   E: ID_MODEL=LUN
   E: ID_REVISION=0.2
   E: ID_SERIAL=360a98000686f6959684a453333524174
   E: ID_TYPE=disk
   E: ID_BUS=scsi
   E: ID_PATH=ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085
   

4. Use the raw and ls commands to confirm raw devices are bound. For example:

   # raw -qa
   /dev/raw/raw1: bound to major 8, minor 17
   /dev/raw/raw2: bound to major 8, minor 65
   
   # ls -l /dev/raw/raw*
   crw-r----- 1 root   oinstall 162, 11 Oct 30 12:54 /dev/raw/raw1
   crw-r----- 1 oracle oinstall 162, 21 Oct 30 14:26 /dev/raw/raw2
   

After you have completed configuring and checking raw storage devices, you can proceed to install Oracle Clusterware and Oracle Real Application Clusters.

4.10.3 Relocate Oracle Clusterware Files from Raw to Block Devices

Oracle recommends that you move Oracle Clusterware files from raw devices to block devices.