G Troubleshooting Disaster Recovery

G.1.1 Heartbeat Failure After Failover in Alert Logs

A warning appears in the alert logs of the database after a failover scenario.

Summary

The following warning appears in the alert logs of the database after a failover scenario, where the new production site database fails to tnsping its remote database instance.

Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_1816.trc:
ORA-16009: remote archive log destination must be a STANDBY database
.
Fri Sep 08 09:11:13 2006
Errors in file c:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_arc1_496.trc:
ORA-16009: remote archive log destination must be a STANDBY database
.
Fri Sep 08 09:11:13 2006
PING[ARC1]: Heartbeat failed to connect to standby 'orcl1_remote1'. Error is 16009.
Fri Sep 08 09:11:50 2006
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[67]: Assigned to RFS process 628
RFS[67]: Database mount ID mismatch [0x4342404d:0x4341ffb0]
Fri Sep 08 09:11:50 2006
Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_628.trc:
ORA-16009: remote archive log destination must be a STANDBY database
.
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[68]: Assigned to RFS process 2488
RFS[68]: Database mount ID mismatch [0x4342404d:0x4341ffb0]
Fri Sep 08 09:12:05 2006
Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_2488.trc:
ORA-16009: remote archive log destination must be a STANDBY database
.
Fri Sep 08 09:12:14 2006
Errors in file c:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_arc1_496.trc:
ORA-16009: remote archive log destination must be a STANDBY database

Steps to Perform

To avoid these error messages in the alert logs, null the log_archive_dest_2 parameter using the following commands:

alter system set log_archive_dest_2='SERVICE=null LGWR ASYNC REOPEN=60';
alter system set log_archive_dest_state_2='defer';

G.1.2 Recommended Method of Patching an Oracle Application Server Disaster Recovery Site

This section describes how to apply an Oracle Application Server patch set to upgrade the Oracle homes that participate in an Oracle Application Server Disaster Recovery site.

Summary

You are unsure how to apply an Oracle Application Server patch set to upgrade the Oracle homes in your Oracle Application Server Disaster Recovery production site.

Steps to Perform

The list in this section describes the steps for applying a patch set to upgrade the Oracle Application Server homes in an Oracle Application Server Disaster Recovery production site.

The following steps assume that the Oracle Central Inventory for any Oracle Application Server instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can b e replicated to the standby site.

Use the following procedure to upgrade Oracle Application Server patch versions:

1. Perform a backup of the production site to ensure that the starting state is secured.

2. Apply the patch set to upgrade the production site instances.

3. After applying the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.

4. After applying the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Some Oracle Application Server patch sets may make updates to repositories, so this step ensures that any changes made to production site databases are synchronized to the standby site databases.

5. The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.

G.1.3 Changing the LockFile Directive for 10.1.2.x and 10.1.3.x Oracle HTTP Server Instances

This manual directs you to install Oracle Application Server instances (including Oracle HTTP Server instances) on shared storage. When the installation of an Oracle home for an Oracle HTTP Server 10.1.2.x or 10.1.3.x instance is performed on shared storage (for example, NAS storage, NFS storage, or SAN storage) that does not provide reliable file locking, Oracle HTTP Server may experience performance problems.

Summary

Some shared storage systems do not provide the reliable file locking that Oracle HTTP Server requires. In these cases, the LockFile directive in the httpd.conf file must be changed to point at a local file system. See the Oracle HTTP Server Administrator's Guide for more information about the LockFile directive.

Steps to Perform

If any 10.1.2.x or 10.1.3.x Oracle HTTP Server instance is installed on shared storage and is experiencing performance problems, perform these steps for the Oracle HTTP Server instance to point the LockFile directive at a local file system:

1. By default, the LockFile directive is commented out. When the Oracle home for an Oracle HTTP Server installation was performed on shared storage (for example, NAS, NFS, or SAN storage), the directive will be in this format:

   #LockFile /<nfs-mounted>/Apache/Apache/logs/httpd.lock
   

2. Edit the $ORACLE_HOME/Apache/Apache/conf/httpd.conf file using the appropriate method for the version of Oracle Application Server you are using.

3. Uncomment the LockFile directive and change it to point at a local file system:

   LockFile /<local_disk>/<path>/httpd.lock
   

4. Restart the Oracle HTTP Server.

After performing these steps, verify that the httpd.lock file exists in the directory specified by the LockFile directive.

G.1.4 Use the DEFAULT_DMS_DIR Environment Variable for Oracle HTTP Server 10.1.3.x Instances

This manual directs you to install Oracle Application Server instances (including Oracle HTTP Server instances) on shared storage. When the installation of an Oracle home for an Oracle HTTP Server 10.1.3.x instance is performed on shared storage (for example, NAS storage, NFS storage, or SAN storage) that does not provide reliable file locking, HTTP Server performance may be poor.

Summary

In Oracle Application Server 10.1.3.x, some new temporary files are created when Oracle HTTP Server is running. These files are created by DMS and are stored in the $ORACLE_HOME/Apache/Apache/logs directory.

When the Oracle home for an Oracle Application Server 10.1.3.x instance is installed on shared storage, the dms_metrics.lock file and other *dms* lock files created under the $ORACLE_HOME/Apache/Apache/logs directory must be moved to a local file system, otherwise Oracle HTTP Server performance may be poor.

Steps to Perform

If any 10.1.2.x or 10.1.3.x Oracle HTTP Server instance is installed on shared storage and is experiencing performance problems, follow these steps to define the DEFAULT_DMS_DIR environment variable and point it to a local file system directory in which to store the *dms* files:

1. Download and apply the Application Server 10.1.3.3.0 patch set to the Oracle home for the Oracle HTTP Server 10.1.3.x instance.

2. Define the DEFAULT_DMS_DIR environment variable in $ORACLE_HOME/opmn/conf/opmn.xml and point it to the local file system directory in which you want to store the *dms* files, for example:

   <environment>
   <variable id="TMP" value="/tmp"/>
   <variable id="DEFAULT_DMS_DIR" value="/tmp"/>
   </environment>
   

3. Stop all the Oracle Application Server processes and start them up again:

   opmnctl stopall
   opmnctl startall
   

4. Verify that the *dms* files have been created under the $DEFAULT_DMS_DIR directory and not under the $ORACLE_HOME/Apache/Apache/logs directory.