This chapter describes the system configuration tasks that you must complete before you start Oracle Universal Installer (OUI) to install Oracle Clusterware.
This chapter contains the following topics:
Overview of Groups and Users for Oracle Clusterware Installations
Requirements for Creating an Oracle Clusterware Home Directory
Note:
Ensure you have minimum hardware configuration completed before installing the operating system.This section provides information about installing a supported Linux distribution. It contains the following topics:
Installing the Oracle Validated RPM From Unbreakable Linux Network
Installing the Oracle Validated RPM From DVD Disks or Images
To complete a minimal Linux installation, select a minimal install option (either a custom installation where you select the Minimal option from Package Group Selection, or where you deselect all packages except for the Base pack). This installation lacks many RPMs required for installation. However, when you install the Oracle Validated RPM for your platform, the RPM downloads the minimum number of packages required to run Oracle Clusterware and Oracle Database.
Unbreakable Linux Network (ULN) customers can obtain the Oracle Validated RPM by using up2date
. If you are not an Unbreakable Linux Network customer, and your operating system is Red Hat or Oracle Linux, then you can obtain the Oracle Validated RPM at the following URLs:
Oracle Linux 4:
http://oss.oracle.com/el4/oracle-validated/
Oracle Linux 5:
http://oss.oracle.com/el5/oracle-validated/
Note:
If you are not a member of Unbreakable Linux Network or RHN (Red Hat support network) and you are a My Oracle Support customer, then you can download instructions to configure a script that replicates Oracle Validated RPM package downloads at the following URL:Search for "minimal Linux".
SSH is required for Oracle Grid Infrastructure installation. OpenSSH should be included in the Linux distribution minimal installation. To confirm that SSH packages are installed, enter the following command:
# rpm -qa |grep ssh
If you do not see a list of SSH packages, then install those packages for your Linux distribution.
Note:
The Oracle Validated RPM installs the X11 client libraries, but does not install the X Window System server packages. To use graphical user interfaces such as OUI, configuration assistants, and Oracle Enterprise Manager, set the display to a system with X Window System server packages.If you do not install the Oracle Validated RPM, then Oracle recommends that you install your Linux operating system with the default software packages (RPMs). This installation includes most of the required packages and helps you limit manual checks of package dependencies. Oracle recommends that you do not customize the RPMs during installation.
For information about a default installation, log on to My Oracle Support:
Search for "Default RPM."
After installation, review system requirements for your distribution to ensure that you have all required kernel packages installed, and complete all other configuration tasks required for your distribution, and for your system configuration.
The Oracle Unbreakable Enterprise Kernel for Linux is available for x86-64 platforms. It is based on a stable 2.6.32 Linux kernel, and also includes optimizations developed in collaboration with Oracle Database, Oracle middleware, and Oracle hardware engineering teams to ensure stability and optimal performance for the most demanding enterprise workloads.
Oracle highly recommends deploying the Oracle Unbreakable Enterprise Kernel in your Linux environment, especially if you are running Oracle software. However, using Oracle Unbreakable Enterprise Kernel is optional. If you require strict Red Hat Enterprise Linux kernel (RHEL) compatibility, then Oracle Linux also includes a kernel compatible with the RHEL Linux kernel, compiled directly from the RHEL source code.
You can obtain more information about the Oracle Unbreakable Enterprise Kernel for Linux at the following URL:
http://www.oracle.com/us/technologies/linux
The Oracle Unbreakable Linux kernel installs directly on top of the Red Hat compatible Oracle Linux 5 or Red Hat Enterprise Linux 5, starting with Update 5, so you are not required to upgrade to a new major release of the operating system to obtain the benefits and features of this new kernel. You can obtain additional information and download the Oracle Unbreakable Enterprise Kernel for Linux at the following URL:
http://public-yum.oracle.com/
The Oracle Unbreakable Enterprise Kernel for Linux is the standard kernel used with Oracle products. The build and QA systems for Oracle Database and other Oracle products use the Oracle Unbreakable Enterprise Kernel for Linux exclusively. The Oracle Unbreakable Enterprise Kernel for Linux is also the kernel used in Oracle Exadata and Oracle Exalogic systems. Oracle Unbreakable Enterprise Kernel for Linux is used in all benchmark tests on Linux in which Oracle participates, as well as in the Oracle Validated Configuration program for x86-64.
If your Linux distribution is Oracle Linux, or Red Hat Enterprise Linux, and you are an Unbreakable Linux customer, then you can complete most preinstallation configuration tasks by using the Oracle Validated Configurations Setup RPM, available from the Unbreakable Linux Network, or available on the Oracle Linux DVD disks.
When it is installed, the Oracle Validated Configuration RPM does the following:
Automatically installs any additional packages needed for installing Oracle Grid Infrastructure and Oracle Database
Creates an oracle
user, and creates the oraInventory (oinstall
) and OSDBA (dba
) groups for that user
Sets and verifies sysctl.conf
settings, system startup parameters, user limits, and driver parameters to values based on recommendations from the Oracle Validated Configurations program
To become an Unbreakable Linux Network customer, contact your sales representative, or purchase a license from the Unbreakable Linux store:
https://shop.oracle.com/product/oraclelinux
To register your server on the Unbreakable Linux Network, or to find out more information, refer to the following URL:
If you are using Oracle Linux 4.7 and higher, or Oracle Linux 5.2 and higher, then the Oracle Validated RPM is included on the install media.
Note:
The Oracle Validated RPM sets kernel parameters and resource limits only for the user accountoracle
. To use multiple software account owners, you must perform system configuration for other accounts manually.
In addition, users and groups are created using the next available ID numbers. If server group and user IDs are not identical on the cluster nodes where you run the Oracle Validated RPM, then it is possible that these ids will be different on the nodes, and the different group and user IDs will cause installation to fail.
To avoid this issue, run the command id
user
on all nodes, where user
is the software owner user account. If group or user IDs are not identical on all the nodes, then change them as necessary to ensure that they are identical.
Use the following procedure to subscribe to Oracle Unbreakable Linux channels, and to add the Oracle Linux channel that distributes the Oracle Validated Configurations Setup RPM:
Complete a default Oracle Linux workstation installation, or a default Red Hat Enterprise Linux installation.
Register your server with Unbreakable Linux Network (ULN). By default, you are registered for the Oracle Linux Latest channel for your operating system and hardware.
Log in to Unbreakable Linux Network at the following URL:
Click the Systems tab, and in the System Profiles list, select a registered server. The System Details window opens, and displays the subscriptions for the server.
From the Available Channels list, select the _base
and _patch
channels corresponding to your Oracle Linux distribution. For example, if your distribution is Oracle Linux 5 Update 5 for x86_64, then select the following:
Oracle Linux 5 Update 5 installation media copy (x86_64)
Oracle Linux 5 Update 5 Patch (x86_64)
Click Subscribe.
From a terminal session, as root
, enter the following command:
# up2date --nox --show-channels
You should see output indicating that you have subscribed to the Oracle Linux channel. For example:
el5_u5_i386_base el5_u5_x86_64_patch
Open a terminal session as root
, and install the Oracle Validated Configurations Setup RPM with up2date
, using the following command:
# up2date --install oracle-validated
Oracle Linux automatically creates a standard (not role-allocated) Oracle installation owner and groups, and sets up other kernel configuration settings as required for Oracle installations.
Repeat steps 1 through 8 on all other servers in your cluster.
Note:
Check the Oracle Validated Configuration RPM log file to review system configuration changes:/var/log/oracle-validated/results/orakernel.log
Use the following procedure to install the Oracle Validated Configuration RPM from the Oracle Linux distribution:
Obtain Oracle Linux disks either by ordering the Oracle Linux Media Pack from Oracle Store, or downloading disk images from the Oracle E-Delivery Web site for Oracle Linux and Oracle VM.
Oracle Store:
https://shop.oracle.com/store/enterpriselinux
E-Delivery Web Site:
http://edelivery.oracle.com/linux
Start the Oracle Linux installation.
At the first software selection screen, which lists task-specific software options, there is an option at the bottom of the screen to customize now or customize later. Select Customize now, and click Next.
On the Customize selection page, select Base System on the list on the left side of the screen, and then select System Tools on the right side of the screen. Then click Optional Packages.
The Packages in System Tools window opens. Select the Oracle Validated RPM package box from the package list, and click Next.
Complete other screens to finish installing Oracle Linux.
Oracle Linux automatically creates a standard (not role-allocated) Oracle installation owner and groups, and sets up other kernel configuration settings as required for Oracle installations.
Repeat steps 2 through 6 on all other cluster member nodes.
If you have an existing Oracle installation, then document version numbers, patches, and other configuration information, and review upgrade procedures for your existing installation. Review Oracle upgrade documentation before proceeding with installation, to decide how you want to proceed.
During rolling upgrades of the operating system, Oracle supports using different operating system binaries when both versions of the operating system are certified with the version of Oracle Database you are running.
Note:
Using mixed operating system versions is only supported for the duration of an upgrade, over the period of a few hours. Oracle does not support operating a cluster with mixed operating systems for an extended period. Oracle does not support running Oracle Clusterware and Oracle Real Application Clusters on heterogeneous platforms (servers with different chip architectures) in the same cluster.To find the most recent updates, and to find best practices recommendations about preupgrade, post-upgrade, compatibility, and interoperability, refer to "Oracle Upgrade Companion." "Oracle Upgrade Companion" is available through Note 466181.1 on OracleMetaLink:
Before you install the Oracle software, you must complete several tasks as the root
user on the system where you install Oracle software. To complete tasks as the root
user on a remote server, enable remote display as root
.
Note:
If you log in as another user (for example,oracle
, or grid
), then repeat this procedure for those users as well.To enable remote display, complete one of the following procedures:
If you are installing the software from an X Window System workstation or X terminal, then:
Start a local terminal session, for example, an X terminal (xterm
).
If you are not installing the software on the local system, then enter a command using the following syntax to enable remote hosts to display X applications on the local X server:
$ xhost + remote_host
where remote_host
is the fully qualified remote hostname. For example:
$ xhost + somehost.example.com somehost.example.com being added to the access control list
If you are not installing the software on the local system, then use the ssh
, command to connect to the system where you want to install the software:
$ ssh remote_host
where remote_host
is the fully qualified remote hostname. For example:
$ ssh somehost.example.com
If you are not logged in as the root
user, then enter the following command to switch the user to root
:
$ su - root password: #
If you are installing the software from a PC or other system with X server software installed, then:
Note:
If necessary, refer to your X server documentation for more information about completing this procedure. Depending on the X server software that you are using, you may need to complete the tasks in a different order.Start the X server software.
Configure the security settings of the X server software to permit remote hosts to display X applications on the local system.
Connect to the remote system where you want to install the software and start a terminal session on that system, for example, an X terminal (xterm
).
If you are not logged in as the root
user on the remote system, then enter the following command to switch user to root
:
$ su - root password: #
You must create the following group and user to install Oracle Clusterware:
The Oracle Inventory group (typically, oinstall
)
You must create this group the first time that you install Oracle software on the system. In Oracle documentation, this group is referred to as oinstall
. Membership in this group controls access to OCR keys, CRS resources, and files and directories in the Oracle Clusterware home that are shared among all Oracle database owners.
Note:
If Oracle software is already installed on the system, then the existing Oracle Inventory group must be the primary group of the operating system user (oracle
or crs
) that you use to install Oracle Clusterware. Refer to "Determining If the Oracle Inventory and Oracle Inventory Group Exists" to identify an existing Oracle Inventory group.Oracle clusterware software owner user (typically, oracle
, if you intend to create a single software owner user for all Oracle software, or crs
, if you intend to create separate Oracle software owners.)
You must create at least one software owner the first time you install Oracle software on the system. This user owns the Oracle binaries of the Oracle Clusterware software, and you can also make this user the owner of the binaries of Automatic Storage Management and Oracle Database or Oracle RAC.
See Also:
Oracle Database Administrator's Reference for UNIX Systems and Oracle Database Administrator's Guide for more information about the OSDBA and OSOPER groups and theSYSDBA
, SYSASM
and SYSOPER
privilegesLog in as root
, and use the following instructions to locate or create the Oracle Inventory group and a software owner for Oracle Clusterware:
Determining If the Oracle Inventory and Oracle Inventory Group Exists
Creating the Oracle Inventory Group If an Oracle Inventory Does Not Exist
Example of Creating the Oracle Clusterware User and OraInventory Path
You must have a group whose members are given access to write to the Oracle Central Inventory (oraInventory
). The Central Inventory contains the following:
A registry of the Oracle home directories (Oracle Clusterware, Oracle Database, and Automatic Storage Management) on the system
Installation logs and trace files from installations of Oracle software. These files are also copied to the respective Oracle homes for future reference
Other metadata inventory information regarding Oracle installations are stored in the individual Oracle home inventory directories, and are separate from the Central Inventory.
The first time you install Oracle software on a system, Oracle Universal Installer checks to see if you have created an Optimal Flexible Architecture (OFA) compliant path in the format u[01-09]/app, such as /u01/app
, and that the user running the installation has permissions to write to that path. If this is true, then Oracle Universal Installer creates the Oracle Inventory directory in the path /u[01-09]/app/oraInventory. For example:
/u01/app/oraInventory
If you have set the environment variable $ORACLE_BASE
for the user performing the Oracle Clusterware installation, then OUI creates the Oracle Inventory directory in the path $ORACLE_BASE/../oraInventory
. For example, if $ORACLE_BASE
is set to /opt/oracle/11
, then the Oracle Inventory directory is created in the path /opt/oracle/oraInventory
.
If you have created neither an OFA-compliant path nor set $ORACLE_BASE
, then the Oracle Inventory directory is placed in the home directory of the user that is performing the installation. For example:
/home/oracle/oraInventory
As this placement can cause permission errors during subsequent installations with multiple Oracle software owners, Oracle recommends that you either create an OFA-compliant installation path, or set an $ORACLE_BASE
environment path.
For new installations, Oracle recommends that you allow OUI to create the Central Inventory directory. By default, if you create an Oracle path in compliance with OFA structure, such as /u01/app
, that is owned by an Oracle software owner, then the Central Inventory is created in the path u01/app/oraInventory
using correct permissions to allow all Oracle installation owners to write to this directory.
When you install Oracle software on the system for the first time, OUI creates the oraInst.loc
file. This file identifies the name of the Oracle Inventory group (typically, oinstall
), and the path of the Oracle Central Inventory directory. An oraInst.loc
file has contents similar to the following:
inventory_loc=central_inventory_location inst_group=group
In the preceding example, central_inventory_location
is the location of the Oracle Central Inventory, and group
is the name of the group that has permissions to write to the central inventory.
If you have an existing Oracle Inventory, then ensure that you use the same Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users you intend to use for installation have permissions to write to this directory.
To determine if you have an Oracle Inventory on your system:
# more /etc/oraInst.loc
If the oraInst.loc
file exists, then the output from this command is similar to the following:
inventory_loc=/u01/app/oracle/oraInventory inst_group=oinstall
In the previous output example:
The inventory_loc
group shows the location of the Oracle Inventory
The inst_group
parameter shows the name of the Oracle Inventory group (in this example, oinstall
).
If the oraInst.loc
file does not exist, then create the Oracle Inventory group by entering a command similar to the following:
# /usr/sbin/groupadd -g 501 oinstall
The preceding command creates the group oinstall
, with the group ID number 501.
You must create a software owner for Oracle Clusterware in the following circumstances:
If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system
If an Oracle software owner user exists, but you want to use a different operating system user, such as crs
, with different group membership, to give separate clusterware and database administrative privileges to those groups in a new Oracle Clusterware and Oracle Database installation.
In Oracle documentation, a user created to own only Oracle Clusterware software installations is called the crs
user. A user created to own either all Oracle installations, or only Oracle database installations, is called the oracle
user.
Note:
If you intend to use multiple Oracle software owners for different Oracle Database homes, then Oracle recommends that you create a separate Oracle software owner for Oracle Clusterware, and install Oracle Clusterware using the Oracle Clusterware software owner.If you want to create separate Oracle software owners (oracle
, crs
, asm
) to create separate users and separate operating system privileges groups for different Oracle software installations, then note that each of these users must have the oraInventory (or oinstall)
group as their primary group, and each user must share the same oraInventory, to prevent corruption of the Central Inventory. Refer to "Creating Custom Configuration Groups and Users for Job Roles".
To determine whether an Oracle software owner user named oracle
or crs
exists, enter a command similar to the following (in this case, to determine if oracle
exists):
# id oracle
If the user exists, then the output from this command is similar to the following:
uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)
Determine whether you want to use the existing user, or create another user.
If you want to use the existing user, then ensure that the user's primary group is the Oracle Inventory group (oinstall
).
If the Oracle software owner (oracle
, crs
) user does not exist, or if you require a new Oracle software owner user, then create it. The following procedure uses crs
as the name of the Oracle software owner.
To create a user, enter a command similar to the following:
# /usr/sbin/useradd -u 501 -g oinstall crs
In the preceding command:
The -u
option specifies the user ID. Using this command flag is optional, as you can allow the system to provide you with an automatically generated user ID number. However, you must make note of the user ID number of the user you create for Oracle Clusterware, as you require it later during preinstallation.
The -g
option specifies the primary group, which must be the Oracle Inventory group. For example: oinstall
.
Use the usermod
command to change user id numbers and groups. For example:
# id oracle uid=500(oracle) gid=500(oracle) groups=500(oracle) # /usr/sbin/usermod -u 500 -g 501 -G 500,502 oracle # id oracle uid=500(oracle) gid=501(oinstall) groups=501(oinstall),500(oracle),502(dba)
Set the password of the user that will own Oracle Clusterware. For example:
# passwd crs
Repeat this procedure on all of the other nodes in the cluster.
The following is an example of how to create the Oracle Clusterware software owner (in this case, crs
), and a path compliant with OFA structure with correct permissions for the oraInventory directory. This example also shows how to create separate Oracle Database and Oracle ASM homes with correct ownership and permissions:
# mkdir -p /u01/app/crs # chown -R crs:oinstall /u01/app # mkdir /u01/app/oracle # chown oracle:oinstall /u01/app/oracle # chmod 775 /u01/app/ # mkdir /u01/app/asm # chown asm:oinstall /u01/app/asm
At the end of this procedure, you will have the following:
/u01
owned by root.
/u01/app
owned by crs:oinstall
with 775 permissions. This ownership and permissions enables OUI to create the oraInventory directory, in the path /u01/app/oraInventory
.
/u01/app/crs
owned by crs:oinstall
with 775 permissions. These permissions are required for installation, and are changed during the installation process.
/u01/app/oracle
owned by oracle:oinstall
with 775 permissions.
/u01/app/asm
owned by asm:oinstall
with 775 permissions.
Each system must meet the following minimum hardware requirements:
Select servers with the same chip architecture; running 32-bit and 64-bit Oracle software versions in the same cluster stack is not supported.
If servers run the same operating system binary, then Oracle Clusterware and Oracle Real Application Clusters support servers with different hardware in the same cluster.
At least 1 GB of physical RAM
Swap space equivalent to the multiple of the available RAM, as indicated in the following table:
Available RAM | Swap Space Required |
---|---|
Between 1 GB and 2 GB | 1.5 times the size of RAM |
Between 2 GB and 8 GB | Equal to the size of RAM |
More than 8 GB | .75 times the size of RAM |
Note:
On Linux, the Hugepages feature allocates non-swappable memory with large page tables using memory-mapped files. If you enable Hugepages, then you should deduct the memory allocated to HugePages from the available RAM before calculating swap space.400 MB of disk space in the /tmp
directory
2 GB of disk space for Oracle Clusterware files, in partitions on separate physical disks, assuming standard redundancy (2 Oracle Cluster Registry partitions and 3 voting disks)
650 MB of disk space for the Oracle Clusterware home
For Linux x86 platforms, if you intend to install Oracle Database, allocate 4 GB of disk space for the Oracle base.
For Linux x86_64 platforms, if you intend to install Oracle Database, allocate 4.6 GB of disk space for the Oracle base.
If you intend to install Oracle Database single instance, allocate between 1.5 and 3.5 GB of disk space for a preconfigured database that uses file system storage. Oracle RAC software only requires 3.2 GB, and Oracle RAC software with sample schemas requires 3.3 GB.
Note:
The volume space requirement for databases that use Automatic Storage Management or shared disk storage is described in Chapter 5.Additional disk space, either on a file system or in an Automatic Storage Management disk group, is required for the Fast recovery area if you choose to configure automated backups.
See Also:
Oracle Database Storage Administrator's GuideThe monitor or display visual on which you run the installation must be capable of displaying a color depth of at least 256 colors.
See Also:
Setting Display and X11 Forwarding Configuration for operating system display configurationTo ensure that each system meets these requirements, follow these steps:
To determine the physical RAM size, enter the following command:
# grep MemTotal /proc/meminfo
If the size of the physical RAM installed in the system is less than the required size, then you must install more memory before continuing.
To determine the size of the configured swap space, enter the following command:
# grep SwapTotal /proc/meminfo
If necessary, refer to your operating system documentation for information about how to configure additional swap space.
To determine the amount of disk space available in the /tmp
directory, enter the following command:
# df -k /tmp
This command displays disk space in 1 kilobyte blocks. On most systems, you can use the df
command with the -h
flag (df -h
) to display output in "human-readable" format, such as "24G" and "10M." If there is less than 400 MB of disk space available in the /tmp
directory (less than 4194304 1-k blocks), then complete one of the following steps:
Delete unnecessary files from the /tmp
directory to make available the disk space required.
Extend the file system that contains the /tmp
directory. If necessary, contact your system administrator for information about extending file systems.
To determine the amount of free RAM and disk swap space on the system, enter the following command:
# free
Note that available RAM and swap space change, depending on user activity on the server.
To determine if the system architecture can run the software, on all platforms, enter the following command:
# uname -m
Note:
This command displays the processor type. If you intend to install on a 64-bit architecture, then the output should be "x86_64."To determine
Review the following sections to check that you have the networking hardware and internet protocol (IP) addresses required for an Oracle Real Application Clusters (Oracle RAC) installation:
Note:
For the most up-to-date information about supported network protocols and hardware for Oracle RAC installations, refer to the Certify pages on the OracleMetaLink Web site at the following URL:https://metalink.oracle.com
The following is a list of requirements for network configuration:
Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).
If you want to use more than one NIC for the public network or for the private network, then Oracle recommends that you use NIC bonding.
The public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes.
For example: With a two-node cluster, you cannot configure network adapters on node1
with eth0
as the public interface, but on node2
have eth1
as the public interface. Public interface names must be the same, so you must configure eth0
as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1
is the private interface for node1
, then eth1
should be the private interface for node2
.
For the public network, each network adapter must support TCP/IP.
For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better required).
Note:
UDP is the default interconnect protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. You must use a switch for the interconnect. Oracle recommends that you use a dedicated switch.Oracle does not support token-rings or crossover cables for the interconnect.
For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test whether an interconnect interface is reachable using a ping
command.
Before starting the installation, you must have the following IP addresses available for each node:
An IP address with an associated host name (or network name) registered in the DNS for the public interface. If you do not have an available DNS, then record the host name and IP address in the system hosts file, /etc/hosts
.
One virtual IP (VIP) address with an associated host name registered in a DNS. If you do not have an available DNS, then record the host name and VIP address in the system hosts file, /etc/hosts
. Select an address for your VIP that meets the following requirements:
The IP address and host name are currently unused (it can be registered in a DNS, but should not be accessible by a ping
command)
The VIP is on the same subnet as your public interface
A private IP address with a host name for each private interface
Oracle recommends that you use private network IP addresses for these interfaces (for example: 10.*.*.* or 192.168.*.*). You can use DNS servers, or the /etc/hosts
file, or both to register the private IP address. Note that if you use DNS servers alone, and the public network becomes unreachable due to NIC or cable failure, then the private IP addresses can fail to resolve.
For the private interconnects, because of Cache Fusion and other traffic between nodes, Oracle strongly recommends using a physically separate, private network. You should ensure that the private IP addresses are reachable only by the cluster member nodes.
During installation, you are asked to identify the planned use for each network interface that OUI detects on your cluster node. You must identify each interface as a public or private interface, and you must use the same private interfaces for both Oracle Clusterware and Oracle RAC.
You can bond separate interfaces to a common interface to provide redundancy, in case of a NIC failure, but Oracle recommends that you do not create separate interfaces for Oracle Clusterware and Oracle RAC. If you use more than one NIC for the private interconnect, then Oracle recommends that you use NIC bonding. Note that multiple private interfaces provide load balancing but not failover, unless bonded.
For example, if you intend to use the interfaces eth2
and eth3
as interconnects, then before installation, you must configure eth2
and eth3
with the private interconnect addresses. If the private interconnect addresses are 10.10.1.1 for eth2
and 10.10.2.1 for eth3
, then bond eth2 and eth3 to an interface, such as bond0
, using a separate subnet such as 10.10.222.0. During installation, define the Oracle Clusterware private node names on 10.10.222.0, and then define 10.10.222.0 (and only that one) as a private interconnect. This ensures that Oracle Clusterware and Oracle RAC are using the same network.
After installation, if you modify interconnects on Oracle RAC with the CLUSTER_INTERCONNECTS
initialization parameter, then you must change it to a private IP address, on a subnet that is not used with a public IP address, nor marked as a public subnet by oifcfg
. Oracle does not support changing the interconnect to an interface using a subnet that you have designated as a public subnet.
See Also:
Oracle Clusterware Administration and Deployment Guide for further information about setting up and using bonded multiple interfacesYou should not use a firewall on the network with the private network IP addresses, as this can block interconnect traffic.
Before installation, check that the default gateway can be accessed by a ping
command. To find the default gateway, use the route
command, as described in your operating system's help utility. After installation, configure clients to use either the VIP address, or the host name associated with the VIP. If a node fails, then the node's virtual IP address fails over to another node.
For example, with a two node cluster where each node has one public and one private interface, you might have the configuration shown in the following table for your network interfaces, where the hosts file is /etc/hosts
:
Node | Host Name | Type | IP Address | Registered In |
---|---|---|---|---|
node1 | node1 | Public | 143.46.43.100 | DNS (if available, else the hosts file) |
node1 | node1-vip | Virtual | 143.46.43.104 | DNS (if available, else the hosts file) |
node1 | node1-priv | Private | 10.0.0.1 | Hosts file |
node2 | node2 | Public | 143.46.43.101 | DNS (if available, else the hosts file) |
node2 | node2-vip | Virtual | 143.46.43.105 | DNS (if available, else the hosts file) |
node2 | node2-priv | Private | 10.0.0.2 | Hosts file |
To enable VIP failover, the configuration shown in the preceding table defines the public and VIP addresses of both nodes on the same subnet, 143.46.43.
Note:
All host names must conform to the RFC 952 standard, which permits alphanumeric characters. Host names using underscores ("_") are not allowed.Before starting the installation, ensure that each member node of the cluster is set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.
On Linux the "-x" flag can be added to the ntpd daemon to prevent the clock from going backwards.
To ensure that your cluster nodes are in synch, and to prevent the clock from going backwards, enter the following command:
service ntpd restart -z
The precise configuration you choose for your network depends on the size and use of the cluster you want to configure, and the level of availability you require.
If certified Network-attached Storage (NAS) is used for Oracle RAC and this storage is connected through Ethernet-based networks, then you must have a third network interface for I/O. Failing to provide three separate interfaces in this case can cause performance and stability problems under load.
For high capacity clusters with a small number of multiprocessor servers, to ensure high availability, you may want to configure redundant network interfaces to prevent a NIC failure from reducing significantly the overall cluster capacity. If you are using network storage, and want to provide redundant network interfaces, then Oracle recommends that you provide six network interfaces: two for the public network interface, two for the private network interface, and two for the network storage.
To verify that each node meets the requirements, follow these steps:
If necessary, install the network adapters for the public and private networks and configure them with either public or private IP addresses.
If you are using a domain name server (DNS), then for each node, register the host names and IP addresses for the public network interfaces in the DNS.
Even if you are using a DNS, Oracle recommends that you add lines to the /etc/hosts
file on each node, specifying the private IP addresses and associated private host names. Oracle also recommends that you add public and virtual IP addresses. Configure the /etc/host file so that it is similar to as shown in the following example, with private interface eth1
, and private hosts nodeint1
and nodeint2
:, where xxx
represents parts of a valid IP address.
#eth0 - PUBLIC xxx.xxx.100.45 node1.example.com node1 xxx.xxx.100.46 node2.example.com node2 #eth1 - PRIVATE 10.0.0.1 nodeint1.example.com nodeint1 10.0.0.2 nodeint2.example.com nodeint2 #VIPs xxx.xxx.100.47 pmvip1.example.com nodevip1 xxx.xxx.100.48 pmvip2.example.com nodevip2
To check network configuration, on each node, enter the following commands:
# /usr/bin/netstat -in
Ensure that each server is properly identified, and that the interface name and IP address for all network adapters that you want to specify as public or private network interfaces are properly configured. In addition, use the ping
command to ensure that each node can obtain a response for the public and private IP addresses from each other node in the cluster.
Note:
When you install Oracle Clusterware and Oracle RAC, you will require the public, private and virtual IP addresses. Make a note of the addresses you configured in the/etc/hosts
file or DNS.To prevent public network failures with Oracle RAC databases using NAS devices or NFS mounts, enter the following command as root to enable the Name Service Cache Daemon (nscd
):
# /sbin/service nscd start
Depending on the products that you intend to install, verify that the following operating system software is installed on the system. To check these requirements refer to the section "Checking the Software Requirements", following this section.
Requirements listed here are current as of the initial release date. To obtain the most current information about kernel requirements, refer to the online version on the Oracle Technology Network (OTN) at the following URL:
http://www.oracle.com/technetwork/indexes/documentation/index.html
For information about current supported Linux distributions, refer to Note ID 169706 on My Oracle Support, which is available from the following URL:
https://support.oracle.com
OUI performs checks your system to verify that it meets the listed operating system package requirements. To ensure that these checks complete successfully, verify the requirements before you start OUI.
Oracle recommends that you install your Linux operating system with the default software packages (RPMs), unless you specifically intend to perform a minimal installation, and follow the directions for performing such an installation to ensure that you have all required packages for Oracle software.
Oracle recommends that you do not customize RPMs during a default operating system installation. A default installation includes most required packages, and will help you to limit manual checks of package dependencies
Note:
Oracle does not support running different operating system versions on cluster members, unless an operating system is being upgraded. You cannot run different operating system version binaries on members of the same cluster, even if each operating system is supported.The following is the list of supported Linux versions and requirements at the time of release:
For installations only of Oracle Clusterware, ensure that you have the kernel versions and packages listed in Table 2-1 and Table 2-2.
If you intend to install Oracle Database or Oracle RAC in addition to Oracle Clusterware, then check Table 2-3 to determine if you must install additional packages for the features you plan to use.
Note:
For Asianux Server, Oracle Linux, and Red Hat Enterprise Linux, system requirements are identical by kernel version. Specifically:Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4 requirements are the same.
Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5 requirements are the same.
Table 2-1 Linux x86 (32-bit) Operating System Kernel Requirements
Table 2-2 Linux x86 (32-bit) Oracle Clusterware and Oracle RAC Package Requirements
Item | Requirements |
---|---|
Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4 |
The following packages (or later versions) must be installed: binutils-2.15.92.0.2-18 compat-libstdc++-33-3.2.3-47.3 elfutils-libelf-0.97-5 elfutils-libelf-devel-0.97-5 gcc-3.4.5-2 gcc-c++-3.4.5-2 glibc-2.3.4-2.19 glibc-common-2.3.4-2.19 glibc-devel-2.3.4-2.19 glibc-headers-2.3.4-2.19 libaio-devel-0.3.105-2 libaio-0.3.105-2 libgcc-3.4.5 libstdc++-3.4.5-2 libstdc++-devel-3.4.5-2 make-3.80-5 sysstat-5.0.5 unixODBC-2.2.11 unixODBC-devel-2.2.11 |
Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5 |
The following packages (or later versions) must be installed: binutils-2.17.50.0.6-2.el5 compat-libstdc++-33-3.2.3-61 elfutils-libelf-0.125 elfutils-libelf-devel-0.125 glibc-2.5-12 glibc-common-2.5-12 glibc-devel-2.5-12 glibc-headers-2.3.4-2 gcc-4.1.1-52 gcc-c++-4.1.1-52 libaio-0.3.106 libaio-devel-0.3.106 libgcc-4.1.1-52 libstdc++-4.1.1 libstdc++-devel-4.1.1-52.e15 make-3.81-1.1 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-devel-2.2.11 |
SUSE 10 Packages |
The following packages (or later versions) must be installed: binutils-2.16.91.0.5 compat-libstdc++-5.0.7 glibc-2.4-31.63 glibc-devel-2.4-31.2 gcc-4.1.0 ksh-93r-12.9 libaio-0.3.104 libaio-devel-0.3.104 libelf-0.8.5 libgcc-4.1.0 libstdc++-4.1.0 libstdc++-devel-4.1.0 make-3.80 sysstat-6.0.2 unixODBC-2.2.11 unixODBC-devel-2.2.11 |
Table 2-3 Linux x86 Oracle Database Features Package Requirements
Item | Requirement |
---|---|
LDAP package |
If you did not perform a default Linux installation, you intend to use LDAP, and you want to use the scripts |
Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer's Kit (XDK) |
Intel C++ Compiler 9.1 or later and the version of GNU C and C++ compilers listed previously for the distribution are supported for use with these products. Note: Intel C++ Compiler v9.1 can be used only with gcc 3.4.5, gcc 4.0 or gcc 4.1 standard template libraries to build OCCI applications. Oracle XML Developer's Kit is supported with the same compilers as OCCI. |
Oracle ODBC Drivers |
If you intend to use Open Database Connectivity (ODBC), then you should install the most recent ODBC Driver Manager for Linux. You should obtain the most current ODBC driver from your operating system vendor. You can read about ODBC at the following URL: http://www.unixodbc.org The ODBC RPMs are only needed if you plan on using ODBC. If you do not plan to use ODBC, then you do not need to install the ODBC RPMs for Oracle Clusterware, Oracle ASM, or Oracle RAC. |
Oracle JDBC/OCI Drivers |
You can use the following optional JDK version with the Oracle JDBC/OCI drivers; however, it is not required for the installation:
Note: By default, IBM Java 5.0 (32-bit) is installed with this release. |
Oracle Real Application Clusters |
For a cluster file system, use one of the following options: OCFS2
For information about OCFS2, refer to the following Web site: http://oss.oracle.com/projects/ocfs2/ For OCFS2 certification status, refer to the Certify page on OracleMetaLink. |
For installations only of Oracle Clusterware, ensure that you have the kernel versions and packages listed in Table 2-4 and Table 2-5.
If you intend to install Oracle Database or Oracle RAC in addition to Oracle Clusterware, then check Table 2-6 to determine if you must install additional packages for the features you plan to use.
Note:
For Asianux Server, Oracle Linux, and Red Hat Enterprise Linux, system requirements are identical by kernel version. Specifically:Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4 requirements are the same.
Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5 requirements are the same.
Oracle Unbreakable Enterprise Kernel for Linux 5 Update 5 (2.6.32), available for x86-64 systems, contains several additional features and performance enhancements not available either with Oracle Linux or with other supported Linux distributions.
Table 2-4 Linux x86-64 Operating System Kernel Requirements
Table 2-5 Linux x86-64 Oracle Clusterware and Oracle RAC Package Requirements
Item | Requirements |
---|---|
Oracle Linux 5 Update 5 with Unbreakable Enterprise Kernel (2.6.32-100.0.19 or later) |
Subscribe to the Oracle Linux 5 channel on the Unbreakable Linux Network, and then install the Oracle Validated RPM. This installs the Oracle Unbreakable Linux kernel, and all required kernel packages for Oracle Grid Infrastructure and Oracle Database installations. |
Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4 |
The following packages (or later versions) must be installed: binutils-2.15.92.0.2 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.97 elfutils-libelf-devel-0.97 gcc-3.4.5 gcc-c++-3.4.5 glibc-2.3.4-2.19 glibc-2.3.4-2.19 (32 bit) glibc-common-2.3.4 glibc-devel-2.3.4 glibc-devel-2.3.4 (32-bit) libaio-0.3.105 libaio-0.3.105 (32 bit) libaio-devel-0.3.105 libgcc-3.4.5 libgcc-3.4.5 (32-bit) libstdc++-3.4.5 libstdc++-3.4.5 (32 bit) libstdc++-devel 3.4.5 make-3.80 sysstat-5.0.5 |
Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5 |
The following packages (or later versions) must be installed: binutils-2.17.50.0.6 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.125 elfutils-libelf-devel-0.125 gcc-4.1.1 gcc-c++-4.1.1 glibc-2.5-12 glibc-2.5-12 (32 bit) glibc-common-2.5 glibc-devel-2.5 glibc-devel-2.5-12 (32 bit) libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libgcc-4.1.1 libgcc-4.1.1 (32 bit) libstdc++-4.1.1 libstdc++-4.1.1 (32 bit) libstdc++-devel 4.1.1 make-3.81 sysstat-7.0.0 |
SUSE 10 |
The following packages (or later versions) must be installed: binutils-2.16.91.0.5 compat-libstdc++-5.0.7-22.2 gcc-4.1.0 gcc-c++-4.1.0 glibc-2.4-31.63 glibc-32bit-2.4-31.2 (32 bit) glibc-devel-2.4 glibc-devel-32bit-2.4 (32 bit) libaio-0.3.104 libaio-32bit-0.3.104 (32 bit) libelf-0.8.5 libgcc-4.1.0 libstdc++-4.1.0 libstdc++-devel-4.1.0 make-3.80 sysstat-6.0.2 |
Table 2-6 Linux x86-64 Oracle Database Features Package Requirements
Item | Requirement |
---|---|
LDAP package |
If you did not perform a default Linux installation, you intend to use LDAP, and you want to use the scripts |
Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer's Kit (XDK) |
Intel C++ Compiler 9.1 or later and the version of GNU C and C++ compilers listed previously for the distribution are supported for use with these products. Note: Intel C++ Compiler v9.1 can be used only with gcc 3.4.5, gcc 4.0 or gcc 4.1 standard template libraries to build OCCI applications. Oracle XML Developer's Kit is supported with the same compilers as OCCI. |
Oracle ODBC Drivers |
If you intend to use Open Database Connectivity (ODBC), then you should install the most recent ODBC Driver Manager for Linux. You should obtain the most current ODBC driver from your operating system vendor. You can read about ODBC at the following URL: http://www.unixodbc.org The ODBC RPMs are only needed if you plan on using ODBC. If you do not plan to use ODBC, then you do not need to install the ODBC RPMs for Oracle Clusterware, Oracle ASM, or Oracle RAC. To use ODBC, you must also install the following additional 32-bit ODBC RPMs, depending on your operating system: Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4: unixODBC-2.2.11 (32 bit) or later unixODBC-devel-2.2.11 (32 bit) or later Asianux Server 3, Oracle Linux 5, and Red Hat Enterprise Linux 5: unixODBC-2.2.11 (32 bit) or later unixODBC-devel-2.2.11 (32 bit) or later SUSE 10: unixODBC-32bit-2.2.11 (32 bit) or later unixODBC-devel-32bit-2.2.11 (32 bit) or later |
Oracle JDBC/OCI Drivers |
You can use the following optional JDK version with the Oracle JDBC/OCI drivers; however, it is not required for the installation:
Note: By default, IBM Java 5.0 (32-bit) is installed with this release. |
Oracle Real Application Clusters |
For a cluster file system, use the following option: OCFS2
For information about OCFS2, refer to the following Web site: http://oss.oracle.com/projects/ocfs2/ For OCFS2 certification status, refer to the Certify page on OracleMetaLink. |
To ensure that the system meets these requirements, follow these steps:
To determine which distribution and version of Linux is installed, enter the following command:
# cat /proc/version
Note:
Only the distributions and versions listed in the previous table are supported. Do not install the software on other versions of Linux.To determine whether the required kernel errata is installed, enter the following command:
# uname -r
The following is sample output displayed by running this command on a Red Hat Enterprise Linux 4.0 system:
2.6.9-55.0.0.0.2.ELsmp
In this example, the output shows the kernel version (2.6.9
) and errata level (55.0.0.0.2.ELsmp
) on the system.
Review the required errata level for your distribution. If the errata level is previous to the required minimum errata update, then obtain and install the latest kernel update from your Linux distributor.
To determine whether the required packages are installed, enter commands similar to the following:
# rpm -q package_name
Alternatively, if you require specific system architecture information, then enter the following command:
# rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep package_name
If a package is not installed, then install it from your Linux distribution media or download the required package version from your Linux distributor's Web site.
To determine if OCFS2 is installed, enter the following command:
# /sbin/modinfo ocfs2
If you want to install the Oracle Database files on an OCFS2 file system and the packages are not installed, then download them from the following Web site. Follow the instructions listed with the kit to install the packages and configure the file system:
http://oss.oracle.com/projects/ocfs2/
Note:
The kernel parameter and shell limit values shown in the following section are recommended values only. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. Refer to your operating system documentation for more information about tuning kernel parameters.On all cluster nodes, verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. The procedure following the table describes how to verify and set the values.
Note:
If the current value for any parameter is greater than the value listed in this table, then do not change the value of that parameter.To view the current value specified for these kernel parameters, and to change them if necessary, follow these steps:
Enter the commands shown in the following table to view the current values of the kernel parameters:
Note:
Make a note of the current values and identify any values that you must change.Parameter | Command |
---|---|
semmsl , semmns , semopm , and semmni |
# /sbin/sysctl -a | grep sem
This command displays the value of the semaphore parameters in the order listed. For example: 250 32000 100 128 |
shmall , shmmax , and shmmni |
# /sbin/sysctl -a | grep shm
For example: kernel.shmmni = 4096 kernel.shmall = 2097152 kernel.shmmax = 2147483648 |
file-max , aio-max |
# /sbin/sysctl -a | grep file-max
For example: fs.file-max = 6815744 fs.aio-max-nr=1048576 |
ip_local_port_range |
# /sbin/sysctl -a | grep ip_local_port_range
This command displays a range of port numbers. For example: net.ipv4.ip_local_port_range = 1024 65000 |
rmem_default , rmem_max , wmem_default , and wmem_max |
# /sbin/sysctl -a | grep net.core.rmem
For example: net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 |
If the value of any kernel parameter is different from the recommended value, then complete the following process:
Using any text editor, create or edit the /etc/sysctl.conf
file, and add or edit lines similar to the following:
Note:
Include lines only for the kernel parameter values that you want to change. For the semaphore parameters (kernel.sem
), you must specify all four values. However, if any of the current system parameter values are greater than the recommended values, then keep using the larger values.kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 fs.aio-max-nr=1048576 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576
By specifying the values in the /etc/sysctl.conf
file, they persist when you restart the system.
To have these changes take effect immediately so that you do not have to restart the system, enter the following command:
# /sbin/sysctl -p
Enter the command /sbin/sysctl -a
to confirm that the values are set correctly.
On SUSE systems only, enter the following command to cause the system to read the /etc/sysctl.conf
file when it restarts:
# /sbin/chkconfig boot.sysctl on
On SUSE 10 systems only, use a text editor to change the /etc/boot.sysctl
parameter RUN_PARALLEL
flag from yes
to no
.
On SUSE systems only, you must enter the GID of the oinstall
group as the value for the parameter /proc/sys/vm/hugetlb_shm_group
. Doing this grants members of oinstall
a group permission to create shared memory segments.
For example, where the oinstall group GID is 501:
# echo 501 > /proc/sys/vm/hugetlb_shm_group
After running this command, use vi
to add the following text to /etc/sysctl.conf
, and enable the boot.sysctl
script to run on system restart:
vm.hugetlb_shm_group=501
Note:
Only one group can be defined as thevm.hugetlb_shm_group
.After updating the values of kernel parameters in the /etc/sysctl.conf file
, either restart the computer, or run the command sysctl -p
to make the changes in the /etc/sysctl.conf
file available in the active kernel memory.
Repeat steps 1 through 7 (as applicable) on all other nodes in the cluster.
Download and install the operating system package cvuqdisk
. Without cvuqdisk
, Cluster Verification Utility is unable to discover shared disks, and you receive the error message "Package cvuqdisk not installed" when you run Cluster Verification Utility. Use the cvuqdisk
rpm for your hardware (for example, x86_64
, or i386
).
To install the cvuqdisk
RPM, complete the following procedure:
Note:
If you prefer, you can choose to disable Cluster Verification Utility shared disk checks by adding the following line to the fileCRS_home
/cv/admin/cvuconfig
:
CV_RAW_CHECK_ENABLED=FALSE
Locate the cvuqdisk
RPM package, which is in the directory rpm
on the installation media. If you have already installed Oracle Clusterware, then it is located in the directory CRS_home
/rpm
.
Copy the cvuqdisk
package to each node on the cluster. You should ensure that each node is running the same version of Linux.
Log in as root.
Using the following command, check to see if you have an existing version of the cvuqdisk
package:
# rpm -qi cvuqdisk
If you have an existing version, then enter the following command to de-install the existing version:
# rpm -e cvuqdisk
Set the environment variable CVUQDISK_GRP
to point to the group that will own cvuqdisk, typically oinstall
. For example:
# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
In the directory where you have saved the cvuqdisk rpm, use the following command to install the cvuqdisk
package:
# rpm -iv cvuqdisk-1.0.1-1.rpm
Before you install and use Oracle Clusterware, you must configure secure shell (SSH) on all cluster nodes for the user that you plan to use to install Oracle Clusterware. In the examples that follow, the Oracle software owner listed is the crs
user.
If you intend to install Oracle RAC or other Oracle software, then you should configure SSH for each of the other users (oracle
, asm
or other software owner) that you plan to use to install software, and either do not use a passphrase, or ensure that you and other installation users load SSH keys into memory before running the installation. As you perform this procedure, replace the example with the user name for which you are configuring SSH.
OUI uses the ssh
and scp
commands during installation to run remote commands on and copy files to the other cluster nodes. You must configure SSH so that these commands do not prompt for a password.
The SSH configuration procedure in this section describes how to configure SSH using SSH1. If SSH is not available, then OUI attempts to use rsh and rcp instead. However, these services are disabled by default on most Linux systems.
This section contains the following:
To determine if SSH is running, enter the following command:
$ pgrep sshd
If SSH is running, then the response to this command is one or more process ID numbers. In the home directory of the software owner that you want to use for the installation (crs
, oracle
), use the command ls -al
to ensure that the .ssh
directory is owned and writable only by the user.
You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while DSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. The instructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation to configure SSH1 compatibility or to configure SSH2 with DSA.
To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (oracle
, crs
, asm
), as SSH ignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keys into memory. In the examples that follow, the RSA key is used.
You must configure SSH separately for each Oracle software installation owner that you intend to use for installation.
To configure SSH, complete the following:
Complete the following steps on each node:
Log in as the software owner (in this example, the crs
user).
To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID you have assigned to the Oracle user, enter the commands id
and id crs
. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical. For example:
$ id uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs) $ id crs uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)
If necessary, create the .ssh
directory in the crs
user's home directory, and set permissions on it to ensure that only the oracle user has read and write permissions:
$ mkdir ~/.ssh $ chmod 700 ~/.ssh
Enter the following command:
$ /usr/bin/ssh-keygen -t rsa
At the prompts:
Accept the default location for the key file (press Enter).
Enter and confirm a pass phrase unique for this installation user.
This command writes the RSA public key to the ~/.ssh/id_rsa.pub
file and the private key to the ~/.ssh/id_rsa
file.
Never distribute the private key to anyone not authorized to perform Oracle software installations.
Repeat steps 1 through 4 on each node that you intend to make a member of the cluster, using the RSA key.
Complete the following steps:
On the local node, change directories to the .ssh
directory in the Oracle Clusterware owner's home directory (typically, either crs
or oracle
).
Then, add the RSA key to the authorized_keys
file using the following commands:
$ cat id_rsa.pub >> authorized_keys $ ls
In the .ssh directory, you should see the id_rsa.pub
keys that you have created, and the file authorized_keys
.
On the local node, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys
file to the oracle
user .ssh directory on a remote node. The following example is with SCP, on a node called node2, with the Oracle Clusterware owner crs
, where the crs
user path is /home/crs
:
[crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/
You are prompted to accept an RSA key. Enter Yes, and you see that the node you are copying to is added to the known_hosts
file.
When prompted, provide the password for the crs
user, which should be the same on all nodes in the cluster. The authorized_keys
file is copied to the remote node.
Your output should be similar to the following, where xxx
represents parts of a valid IP address:
[crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/ The authenticity of host 'node2 (xxx.xxx.173.152) can't be established. RSA key fingerprint is 7e:60:60:ae:40:40:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node1,xxx.xxx.173.152' (RSA) to the list of known hosts crs@node2's password: authorized_keys 100% 828 7.5MB/s 00:00
Using SSH, log in to the node where you copied the authorized_keys
file, using the pass phrase you created. Then change to the .ssh
directory, and using the cat
command, add the RSA keys for the second node to the authorized_keys
file:
[crs@node1 .ssh]$ ssh node2 The authenticity of host node2 (xxx.xxx.100.102) can't be established. RSA key fingerprint is z3:z3:33:z3:z3:33:zz:76:z3:z3:z3. Are you sure you want to continue connecting? (yes/no)? yes Enter passphrase for key '/home/oracle/.ssh/id_rsa': [crs@node2 crs]$ cd .ssh [crs@node2 ssh]$ cat id_rsa.pub >> authorized_keys
Repeat steps 2 and 3 from each node to each other member node in the cluster.
When you have added keys from each cluster node member to the authorized_keys
file on the last node you want to have as a cluster node member, then use scp
to copy the authorized_keys
file with the keys from all nodes back to each cluster node member, overwriting the existing version on the other nodes.
If you want to confirm that you have all nodes in the authorized_keys
file, enter the command more authorized_keys
, and check to see that there is an RSA key for each member node. The file lists the type of key (ssh-rsa), followed by the key, and then followed by the user and server. For example:
ssh-rsa AAAABBBB . . . = crs@node1
Note:
Thecrs
user's /.ssh/authorized_keys
file on every node must contain the contents from all of the /.ssh/id_rsa.pub
files that you generated on all cluster nodes.After you have copied the authorized_keys
file that contains all keys to each node in the cluster, complete the following procedure, in the order listed. In this example, the Oracle Clusterware software owner is named crs
:
On the system where you want to run OUI, log in as the crs
user.
Use the following command syntax, where hostname1
, hostname2
, and so on, are the public hostnames (alias and fully qualified domain name) of nodes in the cluster to run SSH from the local node to each node, including from the local node to itself, and from each node to each other node:
[crs@nodename]$ ssh hostname1 date [crs@nodename]$ ssh hostname2 date . . .
For example:
[crs@node1 crs]$ ssh node1 date The authenticity of host 'node1 (xxx.xxx.100.101)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node1,xxx.xxx.100.101' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon Dec 4 11:08:13 PST 2006 [crs@node1 crs]$ ssh node1.example.com date The authenticity of host 'node1.example.com (xxx.xxx.100.101)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node1.example.com,xxx.xxx.100.101' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon Dec 4 11:08:13 PST 2006 [crs@node1 crs]$ ssh node2 date Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon Dec 4 11:08:35 PST 2006 . . .
At the end of this process, the public hostname for each member node should be registered in the known_hosts
file for all other cluster member nodes.
If you are using a remote client to connect to the local node, and you see a message similar to "Warning: No xauth data; using fake authentication data for X11 forwarding," then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this issue, proceed to "Setting Display and X11 Forwarding Configuration".
Repeat step 2 on each cluster node member.
On each node, enter the following commands to start the SSH agent, and to load the SSH keys into memory:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add
At the prompt, enter the pass phrase for each key that you generated.
For example:
[crs@node1 .ssh]$ exec /usr/bin/ssh-agent $SHELL [crs@node1 .ssh]$ ssh-add Enter passphrase for /home/crs/.ssh/id_rsa Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)
These commands start the ssh-agent on the node, and load the RSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands
If you have configured SSH correctly, then you can now use the ssh
or scp
commands without being prompted for a password or a pass phrase. For example:
[crs@node1 ~]$ ssh node2 date Mon Feb 26 23:34:42 UTC 2007 [crs@node1 ~]$ ssh node1 date Mon Feb 26 23:34:48 UTC 2007 [crs@node1 ~]$ ssh node2
If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys
file on that node contains the correct public keys, and that you have created an Oracle software owner with identical group membership and IDs.
Note:
You must run OUI from this session, or make a note of your SSH pass phrase, and remember to repeat step 4 before you start OUI from a different terminal session.If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:
Bourne, Korn, and Bash shells
$ export DISPLAY=hostname:0
C shell:
$ setenv DISPLAY hostname:0
For example, if you are using the Bash shell, and if your hostname is node1
, then enter the following command:
$ export DISPLAY=node1:0
To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:
Using any text editor, edit or create the software installation owner's ~/.ssh/config
file.
Make sure that the ForwardX11 attribute is set to no
. For example:
Host * ForwardX11 no
During an Oracle Clusterware installation, OUI uses SSH to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc
or .cshrc
) will cause makefile and other installation errors if they contain stty
commands.
To avoid this problem, you must modify these files in each Oracle installation owner user home directory to suppress all output on STDERR, as in the following examples:
Bourne, Bash, or Korn shell:
if [ -t 0 ]; then stty intr ^C fi
C shell:
test -t 0 if ($status == 0) then stty intr ^C endif
Note:
When SSH is not available, the Installer uses thersh
and rcp
commands instead of ssh
and scp
.
If there are hidden files that contain stty
commands that are loaded by the remote shell, then OUI indicates an error and stops the installation.
You run OUI from the user account that you want to own the Oracle Clusterware installation (oracle
or crs
). However, before you start OUI you must configure the environment of the user performing the Oracle Clusterware installation. In addition, create other required Oracle software owners, if needed.
This section contains the following topics:
Environment Requirements for Oracle Clusterware Software Owner
Procedure for Configuring Oracle Software Owner Environments
Setting Shell Limits for the Oracle Software Installation Users
You must make the following changes to configure the Oracle Clusterware software owner environment:
Set the installation software owner user (crs
, oracle
, asm
) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.
Set ulimit settings for file descriptors and processes for the installation software owner (crs
, oracle
)
Set the software owner's environment variable DISPLAY environment variables in preparation for the Oracle Clusterware installation
To set the Oracle software owners' environments, follow these steps, for each software owner (crs
, oracle
, asm
):
Start a new terminal session; for example, start an X terminal (xterm
).
Enter the following command to ensure that X Window applications can display on this system:
$ xhost + hostname
The hostname is the name of the local host.
If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.
If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the crs
user:
$ su - crs
To determine the default shell for the user, enter the following command:
$ echo $SHELL
Open the user's shell startup file in any text editor:
Bash shell (bash):
$ vi .bash_profile
Bourne shell (sh
) or Korn shell (ksh
):
$ vi .profile
C shell (csh
or tcsh
):
% vi .login
Enter or edit the following line, specifying a value of 022 for the default file mode creation mask:
umask 022
If the ORACLE_SID
, ORACLE_HOME
, or ORACLE_BASE
environment variable is set in the file, then remove the appropriate lines from the file.
Save the file, and exit from the text editor.
To run the shell startup script, enter one of the following commands:
Bash shell:
$ . ./.bash_profile
Bourne, Bash, or Korn shell:
$ . ./.profile
C shell:
% source ./.login
If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:
Bourne, Bash, or Korn shell:
$ DISPLAY=local_host:0.0 ; export DISPLAY
C shell:
% setenv DISPLAY local_host:0.0
In this example, local_host
is the host name or IP address of the system that you want to use to display OUI (your workstation or PC).
If you determined that the /tmp
directory has less than 400 MB of free disk space, then identify a file system with at least 400 MB of free space and set the TEMP
and TMPDIR
environment variables to specify a temporary directory on this file system:
Note:
You cannot use a shared file system as the location of the temporary file directory (typically/tmp
) for Oracle RAC installation. If you place /tmp
on a shared file system, then the installation fails.Use the df -h
command to identify a suitable file system with sufficient free space.
If necessary, enter commands similar to the following to create a temporary directory on the file system that you identified, and set the appropriate permissions on the directory:
$ su - root # mkdir /mount_point/tmp # chmod 775 /mount_point/tmp # exit
Enter commands similar to the following to set the TEMP and TMPDIR environment variables:
Bourne, Bash, or Korn shell:
$ TEMP=/mount_point/tmp $ TMPDIR=/mount_point/tmp $ export TEMP TMPDIR
C shell:
% setenv TEMP /mount_point/tmp % setenv TMPDIR /mount_point/tmp
To verify that the environment has been set correctly, enter the following commands:
$ umask $ env | more
Verify that the umask
command displays a value of 22
, 022
, or 0022
and that the environment variables you set in this section have the correct values.
To improve the performance of the software on Linux systems, you must increase the following shell limits for the Oracle software owner users (crs
, oracle
, asm
):
Shell Limit | Item in limits.conf | Hard Limit |
---|---|---|
Maximum number of open file descriptors | nofile |
65536 |
Maximum number of processes available to a single user | nproc |
16384 |
Add the following lines to the /etc/security/limits.conf
file (the following example shows the oracle
user as the software owner):
oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536
Add or edit the following line in the /etc/pam.d/login
file, if it does not already exist:
session required pam_limits.so
Depending on your shell environment, make the following changes to the default shell startup file (note that these examples show the user oracle
):
For the Bourne, Bash, or Korn shell, add lines similar to the following to the /etc/profile
file (or the file on SUSE systems)/etc/profile.local
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi if [ $USER = "root" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi
For the C shell (csh
or tcsh
), add the following lines to the /etc/csh.login
file (or the file on SUSE systems)/etc/csh.login.local
:
if ( $USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif if ( $USER == "root" ) then limit maxproc 16384 limit descriptors 65536 endif
Repeat this procedure on all other nodes in the cluster, and for all Oracle software owners that you intend to use to install Oracle software (crs
, asm
, oracle
).
During installation, you are prompted to provide a path to a home directory to store Oracle Clusterware binaries. Ensure that the directory path you provide meets the following requirements:
It should be created in a path separate from existing Oracle homes.
It should not be located in a user home directory.
It should be created either as a subdirectory in a path where all files can be owned by root, or in a unique path.
Before installation, it should be owned by the installation owner of Oracle Clusterware (typically oracle
for a single installation owner for all Oracle software, or crs
for role-based Oracle installation owners), and set to 750 permissions.
For installations with Oracle Clusterware only, Oracle recommends that you create a path compliant with Oracle Optimal Flexible Architecture (OFA) guidelines, so that Oracle Universal Installer (OUI) can select that directory during installation. For OUI to recognize the path as an Oracle software path, it must be in the form u0[1-9]/app.
When OUI finds an OFA-compliant path, it creates the Oracle Clusterware and Oracle Central Inventory (oraInventory) directories for you.
Create an Oracle Clusterware path. For example:
# mkdir -p /u01/app # chown -R crs:oinstall /u01
Alternatively, if you later intend to install Oracle Database software, then create an Oracle base path. OUI automatically creates an OFA-compliant path for Oracle Clusterware derived from the Oracle base path. The Optimal Flexible Architecture path for the Oracle Base is /u01/app/
user
, where user
is the name of the user account that you want to own the Oracle Database software. For example:
# mkdir -p /u01/app/oracle # chown -R oracle:oinstall /u01/app/oracle # chmod -R 775 /u01/app/oracle
Note:
If you choose to create an Oracle Clusterware home manually, then do not create the Oracle Clusterware home under Oracle base. Creating an Oracle Clusterware installation in an Oracle base directory will cause succeeding Oracle installations to fail.See Also:
"Creating Standard Configuration Operating System Groups and Users", and "Creating Custom Configuration Groups and Users for Job Roles" for information about creating groups, users, and software homes for additional Oracle software installationsCluster Verification Utility (CVU) is a tool that performs system checks. This guide provides Cluster Verification Utility commands to assist you with confirming that your system is properly configured for Oracle Clusterware and Oracle RAC installation.
This section describes the following topics:
Using CVU to Determine if Installation Prerequisites are Complete
Using Cluster Verification Utility with Oracle Database 10g Release 1 or 2
Cluster Verification Utility is provided with two scripts: runcluvfy.sh
, which is designed to be used before installation, and cluvfy
, which is in the path CRS_home
/bin
. The script runcluvfy.sh
contains temporary variable definitions which enable it to be run before installing Oracle Clusterware or Oracle Database. After you install Oracle Clusterware, use the command cluvfy
to check prerequisites and perform other system readiness checks.
Before Oracle software is installed, to enter a Cluster Verification Utility command, change directories and start runcluvfy.sh
using the following syntax:
cd /mountpoint ./runcluvfy.sh options
In the preceding example, the variable mountpoint represents the mountpoint path for the installation media and the variable options
represents the Cluster Verification Utility command options that you select. For example:
$ cd /mnt/dvdrom $ ./runcluvfy.sh comp nodereach -n node1,node2 -verbose
By default, when you enter a Cluster Verification Utility command, it provides a summary of the test. During preinstallation, Oracle recommends that you obtain detailed output by using the -verbose
argument with the Cluster Verification Utility command. The -verbose
argument produces detailed output of individual checks. Where applicable, it shows results for each node in a tabular layout.
You can use Cluster Verification Utility to determine which system prerequisites for installation are already completed. Use this option if you are installing Oracle 11g release 1 (11.1) on a system with a pre-existing Oracle software installation. In using this option, note the following:
You must complete the prerequisites for using Cluster Verification Utility, notably configuring SSH between all nodes in the cluster, before you can complete a clusterwide status check.
Cluster Verification Utility can assist you by finding preinstallation steps that need to be completed, but it cannot perform preinstallation tasks
Use the following syntax to determine what preinstallation steps are completed, and what preinstallation steps must be performed
$ ./runcluvfy.sh stage -pre crsinst -n node_list
In the preceding syntax example, replace the variable node_list
with the names of the nodes in your cluster, separated by commas.
For example, for a cluster with mountpoint /mnt/dvdrom/
, and with nodes node1
, node2
, and node3
, enter the following command:
$ cd /mnt/dvdrom/ $ ./runcluvfy.sh stage -pre crsinst -n node1,node2,node3
Review the Cluster Verification Utility report, and proceed to the sections of the preinstallation chapter to complete additional steps as needed.
The cluvfy
commands have context-sensitive help that shows correct syntax usage based on the command line arguments that you enter.
If you enter an invalid Cluster Verification Utility command, then it shows the correct usage for that command. For example, if you type runcluvfy.sh stage -pre dbinst
, then Cluster Verification Utility shows the correct syntax for the database preinstallation checks that Cluster Verification Utility performs with the dbinst
stage option. The following is a list of context help commands.
cluvfy -help
: Cluster Verification Utility displays detailed Cluster Verification Utility command information.
cluvfy comp -list
: Cluster Verification Utility displays a list of components that can be checked, and brief descriptions of how each component is checked.
cluvfy comp -help
: Cluster Verification Utility displays detailed syntax for each of the valid component checks.
cluvfy stage -list
: Cluster Verification Utility displays a list of valid stages.
cluvfy stage -help
: Cluster Verification Utility displays detailed syntax for each of the valid stage checks.
You can use Cluster Verification Utility on the Oracle Database 11g release 1 (11.1) media to check system requirements for Oracle Database 10g release 1 (10.1) and later installations. To use Cluster Verification Utility to check 10. 2 installations, append the command flag -r 10gR2
to the standard Cluster Verification Utility system check commands.
For example, to perform a verification check for a Cluster Ready Services 10. 2 installation, on a system where the media mountpoint is /mnt/dvdrom
, and the cluster nodes are node1
, node2
, and node3
, enter the following command:
$ cd /mnt/dvdrom $ ./runcluvfy.sh stage -pre crsinst -n node1,node2,node3 -r 10gR2
Note:
If you do not specify a release version to check, then Cluster Verification Utility checks for 11g release 1 (11.1) requirements.If you run Cluster Verification Utility using the -verbose
argument, and a Cluster Verification Utility command responds with UNKNOWN
for a particular node, then this is because Cluster Verification Utility cannot determine if a check passed or failed. The following is a list of possible causes for an "Unknown" response:
The node is down
Executables required by Cluster Verification Utility are missing in the /bin
directory in the Oracle Clusterware home or Oracle home directory
The user account starting Cluster Verification Utility does not have privileges to run common operating system executables on the node
The node is missing an operating system patch, or a required package
The node has exceeded the maximum number of processes or maximum number of open files, or there is a problem with IPC segments, such as shared memory or semaphores
Use Cluster Verification Utility (CVU) to check your servers for their readiness to install Oracle Clusterware:
As the installation owner user (oracle
or crs
), ensure that you have ssh keys loaded into memory, and enter a command using the following syntax to verify that your cluster is properly configured for Oracle Clusterware installation:
/mountpoint/runcluvfy.sh stage -pre crsinst -n node_list [-verbose]
In the preceding syntax example, the variable node_list
is a comma-delimited list of nodes in your cluster. This command checks node reachability, user and group equivalence on each node, node connectivity, and basic system requirements, including kernel versions and packages.
Select the option -verbose
to receive progress updates as Cluster Verification Utility performs its system checks, and detailed reporting of the test results.
For example, to verify system readiness on a two-node cluster with nodes node1
and node2
, with the mountpoint /mnt/dvdrom
, and with updates and a summary of the verification checks Cluster Verification Utility performs, enter the following command:
$ /mnt/dvdrom/runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose