9 Backup, Recovery, and Disaster Recovery

Repository Backup

Oracle recommends using High Availability Best Practices for protecting the Repository database against unplanned outages. As such, use the following standard database backup strategies.

• Database should be in archivelog mode. Not running the repository database in archivelog mode leaves the database vulnerable to being in an unrecoverable condition after a media failure.

• Perform regular hot backups with RMAN using the Recommended Backup Strategy option via the Enterprise Manager console. Other utilities such as DataGuard and RAC can also be used as part of a comprehensive strategy to prevent data loss.

Adhering to these strategies will create a full backup and then create incremental backups on each subsequent run. The incremental changes will then be rolled up into the baseline, creating a new full backup baseline.

Using the Recommended Backup Strategy also takes advantage of the capabilities of Enterprise Manager to execute the backups: Jobs will be automatically scheduled through the Job sub-system of Enterprise Manager. The history of the backups will then be available for review and the status of the backup will be displayed on the repository database target home page. This backup job along with archiving and flashback technologies will provide a restore point in the event of the loss of any part of the repository. This type of backup, along with archive and online logs, allows the repository to be recovered to the last completed transaction.

Setting Up the Backup

First, navigate to the Enterprise Manager Recovery Settings page (Target-->Database--><Repository Database Target>-->Availability-->Recovery Settings) and enable Archive Logging then Flashback Database as shown in Figure 9-1.

Figure 9-1 Recovery Settings Page

Description of "Figure 9-1 Recovery Settings Page"

Next, navigate to the Backup Policies page (Target-->Database--><Repository Database Target>-->Availability-->Backup Settings-->Policy) and enable Block Change Tracking to speed up backup operations as shown in Figure 9-2.

Figure 9-2 Backup Policy Page

Description of "Figure 9-2 Backup Policy Page"

A thorough summary of how to configure backups using Enterprise Manager is available in the Oracle Database 2 Day DBA guide. For additional information on Database high availability best practices, review the Oracle Database High Availability Best Practices documentation.

Repository Recovery

Recovery of the Repository database must be performed using RMAN since Grid Control will not be available when the repository database is down. There are two recovery cases to consider:

• Full Recovery: No special consideration is required for Enterprise Manager.

• Point in time/Incomplete Recovery: Recovered repository may be out of sync with Agents because of lost transactions. In this situation, some metrics may show up incorrectly in the Grid Control console unless the repository is synchronized with the latest state available on the Agents.

Beginning with Enterprise Manager version 10.2.0.5, a new repository resync feature allows you to automate the process of synchronizing the Enterprise Manager repository with the latest state available on the Agents. To resynchornize the repository with the Agents, you use Enterprise Manager Command-line utility (emctl) resync repos command:

emctl resync repos -full -name "<descriptive name for the operation>"

You must run this command from the OMS Oracle Home after restoring the repository but BEFORE starting the OMS. After submitting the command, start up all OMS's and monitor the progress of repository resychronization from the Enterprise Manager console's Repository Resynchronization page, as shown in Figure 9-3.

Figure 9-3 Repository Synchronization Page

Description of "Figure 9-3 Repository Synchronization Page"

Repository recovery is complete when the resynchronization jobs complete on all Agents.

Oracle strongly recommends that the repository database be run in archivelog mode so that in case of failure, the database can be recovered to the latest transaction. If the database cannot be recovered to the last transaction, Repository Synchronization can be used to restore monitoring capabilities for targets that existed when the last backup was taken. Actions taken after the backup will not be recovered automatically. Some examples of actions that will not be recovered automatically by Repository Synchronization are:

• Notification Rules

• Preferred Credentials

• Groups, Services, Systems

• Jobs/Deployment Procedures

• Custom Reports

• New Agents

Manually Resynchronizing Agents

The Enterprise Manager Repository Synchronization feature can only be used for Agents 10.2.0.5 or later. Older Agents must be resynchronized manually by shutting down the Agent using the following procedure:

1. Shut down the Agent.

2. Delete the agentstmp.txt, lastupld.xml, state/* and upload/* files from the $AGENT_HOME/sysman/emd directory.

3. Restart the Agent.

Recovery Scenarios

A prerequisite for repository (or any database) recovery is to have a valid, consistent backup of the repository. Using Enterprise Manager to automate the backup process ensures regular, up-to-date backups are always available if repository recovery is ever required. Recovery Manager (RMAN) is a utility that backs up, restores, and recovers Oracle Databases. The RMAN recovery job syntax should be saved to a safe location. This you to perform a complete recovery of the Enterprise Manager repository database. In its simplest form, the syntax would appear as follows:

run {

restore database;

recover database;

}

Actual syntax will vary in length and complexity depending on your environment. For more information on extracting syntax from an RMAN backup and recovery job, or using RMAN in general, see the Oracle Database Backup and Recovery Advanced User's Guide.

The following scenarios illustrate various repository recovery situations along with the recovery steps.

Backing Up the OMS

The OMS is generally stateless. Some transient and configuration data is stored on the OMS file system. The shared loader “recv”directory stores metric data uploaded from Agents temporarily before the data is loaded into the repository. If an OMS goes down, other surviving OMS(s) upload the data stored in the shared loader location. In a High Availability (HA) configuration, the shared loader receive directory should be protected using some HA storage technology.

Beginning with Enterprise Manager version 10.2.0.5, a snapshot of OMS configuration can be taken using the emctl exportconfig oms command.

emctl exportconfig oms [-sysman_pwd <sysman password>]

      [-dir <backup dir>]     Specify directory to store backup file

      [-keep_host]            Specify this parameter if the OMS was installed

                              using a virtual hostname. 

                              For example: ORACLE_HOSTNAME

Running exportconfig captures a snapshot of the OMS at a given point in time, thus allowing you to back up the most recent OMS configuration on a regular basis. If required, the most recent snapshot can then be restored on a fresh OMS installation on the same or different host.

Recovering the OMS

If an OMS is lost, it should be reinstalled using “Installing Software-Only and Configuring Later". This is an additional Management Service option documented in the Oracle Enterprise Manager Grid Control Installation and Basic Configuration guide. The OMS configuration can then be restored with the following command:

emctl importconfig oms

Use export file backed up earlier. This command is available beginning with Enterprise Manager version 10.2.0.5.

OMS Recovery Scenarios

The following scenarios illustrate various OMS recovery situations along with the recovery steps.

Backing Up Agents

There are no special considerations for backing up Agents. As a best practice, reference Agent installs should be maintained for different platforms and kept up-to-date in terms of customizations in the emd.properties file and patches applied. Use Deployment options from the Grid Control console to install and maintain Reference Agent installs.

Recovering Agents

If an Agent is lost, it should be reinstalled by cloning from a reference install. Cloning from a reference install is often the fastest way to recover an Agent install as one does not have to track and reapply customizations and patches. Care should be taken to reinstall the Agent using the same port. Using the Enterprise Manager's Agent Resynchronization feature, a reinstalled Agent can be reconfigured using target information present in the repository. When the Agent is reinstalled using the same port, the OMS detects that it has been re-installed and blocks it temporarily to prevent the auto-discovered targets in the re-installed Agent from overwriting any customizations done previously.

The Agent can be resynchronized and unblocked from the Agent homepage by clicking on the Resynchronize Agent button. Resynchronization pushes all targets from the repository to the Agent and then unblocks the Agent.

Agent Recovery Scenarios

The following scenarios illustrate various Agent recovery situations along with the recovery steps. The Agent resynchronization feature requires that a reinstalled Agent use the same port as the previous Agent that crashed.

Collapsed configuration, recovery on the same host, incomplete recovery of repository

Repository and the OMS are installed on same host (host "A"). No Server Load Balancer is configured. The repository database is running in noarchivelog mode. Full cold backup is available. Export of OMS configuration via emctl exportconfig oms is available. The repository, OMS and the Agent crash.

1. Recover the repository database using RMAN.

2. Since the OMS OracleHome is not available and repository resynchronization has to be initiated before starting an OMS against the restored repository, submit "resync" via the following PL/SQL block. Log into the repository as SYSMAN using SQLplus and run:

   begin emd_maintenance.full_repository_resync('<resync name>'); end;

3. Deinstall the crashed OMS and Agent OracleHomes using OUI.

4. Reinstall the OMS using “Install Software only, configure later”. This is an additional management service option. Location of the install need not be same as previous install.

5. Import the OMS configuration:

   emctl importconfig oms -file <exportfile>

   Because the Agent that is installed with the OMS uses the same port as before, the OMS automatically blocks the Agent. At this point, you must "resync" the Agent from the Agent homepage.

6. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under $AGENT_HOME/sysman/emd directory. Restart the Agent.

   Note:

   he Repository Synchronization function will automatically fix all 10.2.0.5 and later Agents.

7. Log into Enterprise Manager and navigate to the Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the errors.

8. Verify that the site is fully operational.

Distributed configuration, Multi-OMS with SLB, recovery on different hosts, incomplete recovery of repository

Repository installed on host "X". Two OMSs are installed-- one on host "A" and one on host "B". OMSs are fronted by an SLB. Repository database running in noarchivelog mode. Full offline backup available. Host "X" and host "B" are lost.

1. Stop the OMS on host "A" using opmnctl stopall

2. Recover the database using RMAN on new host "Y"

3. Update the connect descriptor in the emoms.properties file on host "A" to point to host "Y".

4. Initiate Repository resynchronization:

   emctl resync repos -full -name "<resync name>"

   from the OMS Oracle home of host "A"

5. Run the command to relocate and reconfigure the repository database target:

   emctl config repos -agent <agent on hostY> -oh <OH on hostY> -host <hostY> -conn_desc "<TNS connect descriptor>"

6. Export latest OMS configuration from host "A":

   emctl exportconfig oms -dir <export location>

7. Ensure that shared loader receive directory and shared software library locations are accessible from the host "C".

8. Start the OMS on hostA using opmnctl startall

9. Reinstall the OMS on host "C" using the “Install Software only, configure later” – Additional management service option. Location of install need not be same as previous install.

10. Import the OMS configuration:

    emctl importconfig oms -file <exportfile>

11. Update the SLB pools by replacing host "B" entries with host "C".

12. Resecure the Agent that gets installed along with OMS on host "C"

    emctl secure agent -emdWalletSrcUrl "http://slb/:<httpport>/em"

13. Relocate and reconfigure the OMS and Repository target:

    emctl config emrep -agent <agent on hostC> -conn_desc "<TNS connect descriptor>"

14. Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from Agent "B" to Agent "C". Delete the old Agent on host "B".

15. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under $AGENT_HOME/sysman/emd directory. Restart the Agent.

16. Login to Grid Control and navigate to the Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any , after fixing any errors mentioned.

17. Verify that the site is fully operational.