Note: This is an archival copy of Security Sun Alert 200810 as previously published on http://sunsolve.sun.com.|
Latest version of this security advisory is available from http://support.oracle.com as Sun Alert 1000611.1.
Sun Cluster 2.2
Sun Cluster 3.0
Sun Cluster 3.1
Sun Cluster 3.2
Date of Resolved Release
TCP Port Conflict Between Sun Cluster for OPS/RAC and Solaris Secure Shell Server.
During a cluster reconfiguration event (e.g. when another node joins or leaves the cluster), a Sun Cluster 3.x node can panic, and a Sun Cluster 2.2 node can abort from the cluster. Also, if a local unprivileged user is allowed to run client applications on a Sun Cluster system, they may be able to run an application which utilizes a TCP port which is also used by the DLM and thus trigger this issue, causing a Denial of Service.Contributing Factors
This issue can occur in the following releases:
This issue can only occur if all of the following are true:
A TCP port conflict may occur for the above Sun Cluster releases which have the OPS/RAC packages installed (specifically the ORCLudlm and SUNWudlm packages) and the Solaris Secure Shell server daemon (sshd(1M)) running on the system configured to allow X11 forwarding. Sun ships Solaris Secure Shell with Solaris 9 only. Sites using earlier versions of Solaris may have installed a third party version of Secure Shell.
It is recommended that a Sun Cluster system should not be used for client applications as described in:
Sun Cluster 3.0 Release Notes
Sun Cluster 2.2 Software Installation Guide
For Sun Cluster 3.x
During an SC3.x cluster reconfiguration event (e.g. when another node joins or leaves the cluster), one or more of the active cluster nodes can panic with the following message:
panic[cpu0]/thread=2a100045d20: Failfast: Aborting because "ucmmd" died 30
Note: This same panic can also occur for a number of other reasons. In this particular case, the system messages file (/var/adm/messages) will contain the relevant message:
Oct 13 15:08:36 vha-2a ID[SUNWudlm.udlm]: Unix DLM initiating cluster abort.
The DLM log file (/var/cluster/ucmm/dlm_<node-name>/logs/dlm.log) will also contain the relevant message:
15:08:36-00000-00620- BIND ERROR: 'Address already in use', family=2, port=6007,
For Sun Cluster 2.2
During an SC2.2 cluster reconfiguration event (e.g. when another node joins or leaves the cluster), one or more of the active cluster nodes can abort from the cluster. The system messages file (/var/adm/messages) will contain the messages:
Nov 4 10:40:03 vha-2a ID[SUNWcluster.udlm.4004]: Unix DLM initiating cluster
The DLM log file (/var/opt/SUNWcluster/dlm_<node-name>/logs/dlm.log) will also contain the relevant message:
10:40:03-00000-00858- BIND ERROR: 'Address already in use', family=2, port=6004,
If Solaris Secure Shell X11 forwarding is not required, disable it by editing the server configuration file on all cluster nodes as follows:
# cd /etc/ssh # cp sshd_config sshd_config.save # vi sshd_config
Then restart the sshd daemon:
# /etc/init.d/sshd stop # /etc/init.d/sshd start
For Solaris 10, restart the sshd daemon as follows:
# svcadm disable network/ssh
If Solaris Secure Shell X11 forwarding is required, the DLM should be reconfigured to use an alternate, unused range of TCP port numbers. The system administrator should use the "/etc/services" file and their knowledge of the TCP applications that run on the cluster nodes when choosing a new port range for the DLM.
In the following three examples (for Sun Cluster versions 3.1 update 1, 3.1/3.0 and 2.2), the DLM's port range is changed to the currently unassigned range of 1124-1154 (inclusive).
Example 1: Sun Cluster version 3.1 update 1
The following procedure for changing extension properties is also described in the Sun Cluster 3.1 Data Service for Oracle Parallel Server/Real Application Clusters Guide, in the section "How to Modify an Extension Property That Is Tunable Only When a Resource Is Disabled".
First, shutdown all Oracle OPS/RAC instances running on all nodes in the cluster.
Disable all RAC framework resources:
# scswitch -n -j rac_cvm # scswitch -n -j rac_udlm # scswitch -n -j rac_hwraid # scswitch -n -j rac_framework
Take the RAC framework resource group offline, and change it to the unmanaged state:
# scswitch -F -g rac-framework-rg # scswtich -u -g rac-framework-rg
Reboot all the nodes in the cluster. During the restart, the "rac-framework-rg" resource group will not be started automatically on any node, which allows you to alter the port range:
# scrgadm -c -j rac_udlm -x Port=1124 -x Num_ports=30
Start the "rac-framework-rg" resource group:
# scswitch -Z -g rac-framework-rg
All Oracle OPS/RAC instances can now be restarted on all nodes in the cluster.
Example 2: Sun Cluster versions 3.0 and 3.1
Reconfigure the DLM's port range by editing the DLM configuration file on all cluster nodes as follows:
# cd /opt/SUNWudlm/etc # cp udlm.conf udlm.conf.save # vi udlm.conf
udlm.port : 6000
udlm.port : 1124
Note: This operation must be performed on ALL nodes in the cluster, and ALL nodes must be configured to use exactly the same port range. When the changes have been made, all cluster nodes must then be rebooted simultaneously for the changes to take effect.
On one node run:
# scshutdown -y -g0
Example 3: Sun Cluster version 2.2
Reconfigure the DLM's port range by editing the cdb configuration file on all cluster nodes as follows:
# cd /etc/opt/SUNWcluster/conf # cp <clustername>.cdb <clustername>.cdb.save # vi <clustername>.cdb
udlm.port : 6000
udlm.port : 1124
Note: This operation must be performed on ALL nodes in the cluster, and ALL nodes must be configured to use exactly the same port range. When the changes have been made, all cluster nodes must then be removed simultaneously from the cluster and then rejoined for the changes to take effect.
On all nodes, run:
# scadmin stopnode
Then on one node:
# scadmin startcluster <nodename> <clustername>
And on the remaining nodes:
# scadmin startnode
The three examples provided above avoid the problem of a port conflict between the DLM and Secure Shell, but do not protect against malicious denial of service attacks from unprivileged users logged into the cluster nodes.
To avoid possible denial of service attacks on the DLM, it's port numbers should be changed to a privileged range (i.e. a range below 1024). Again, the system administrator should use the /etc/services file and their knowledge of the TCP applications that run on the cluster nodes when choosing a new port range for the DLM. The range 918-949 may be suitable for most systems, as it is currently unassigned by the Internet Assigned Numbers Authority (IANA) e.g.
udlm.port : 918Resolution
There is no resolution to this problem incorporated into the issued product(s). Please change the DLM's port range using the methods described above to values that suite your own systems.
This Sun Alert notification is being provided to you on an "AS IS"
basis. This Sun Alert notification may contain information provided by
third parties. The issues described in this Sun Alert notification may
or may not impact your system(s). Sun makes no representations,
warranties, or guarantees as to the information contained herein. ANY
AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR
NON-INFRINGEMENT, ARE HEREBY DISCLAIMED. BY ACCESSING THIS DOCUMENT YOU
ACKNOWLEDGE THAT SUN SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES THAT ARISE OUT
OF YOUR USE OR FAILURE TO USE THE INFORMATION CONTAINED HEREIN. This
Sun Alert notification contains Sun proprietary and confidential
information. It is being provided to you pursuant to the provisions of
your agreement to purchase services from Sun, or, if you do not have
notification may only be used for the purposes contemplated by these
27-Apr-2004: Updated the Relief/Workaround section
13-Aug-2008: Updated Contributing Factors section
04-Sep-2008: Updated Workaround section
This solution has no attachment