3 Grid Control Common Configurations

3.4.2.1 Monitoring the Load on Your Management Service Installations

If your environment is not configured with an SLB, to keep the workload evenly distributed you should always be aware of how many Management Agents are configured per Management Service and monitor the load on each Management Service.

At any time, you can view a list of Management Agents and Management Services using Setup on the Grid Control console.

Use the charts on the Overview page of Management Services and Repository to monitor:

• Loader backlog (files)

  The Loader is part of the Management Service that pushes metric data into the Management Repository at periodic intervals. If the Loader Backlog chart indicates that the backlog is high and Loader output is low, there is data pending load, which may indicate a system bottleneck or the need for another Management Service. The chart shows the total backlog of files totaled over all Oracle Management Services for the past 24 hours. Click the image to display loader backlog charts for each individual Management Service over the past 24 hours.

• Notification delivery backlog

  The Notification Delivery Backlog chart displays the number of notifications to be delivered that could not be processed in the time allocated. Notifications are delivered by the Management Services. This number is summed across all Management Services and is sampled every 10 minutes. The graph displays the data for the last 24 hours. It is useful for determining a growing backlog of notifications. When this graph shows constant growth over the past 24 hours, then consider adding another Management Service, reducing the number of notification rules, and verifying that all rules and notification methods are useful and valid.

3.4.2.2 Monitoring the Response Time of the Enterprise Manager Web Application Target

The information on the Management Services and Repository page can help you determine the load being placed on your Management Service installations. More importantly, consider how the performance of your Management Service installations is affecting the performance of the Grid Control Console.

Use the EM Website Web Application target to review the response time of the Grid Control Console pages:

1. From the Grid Control Console, click the Targets tab and then click the Web Applications subtab.

2. Click EM Website in the list of Web Application targets.

3. In the Key Test Summary table, click homepage. The resulting page provides the response time of the Grid Control Console homepage URL.

   See Also:

   The Enterprise Manager online help for more information about using the homepage URL and Application Performance Management (also known as Application Performance Monitoring) to determine the performance of your Web Applications

4. Click Page Performance to view the response time of some selected Grid Control Console pages.

   Note:

   The Page Performance page provides data generated only by users who access the Grid Control Console by way of the OracleAS Web Cache port (usually, 7777).

5. Select 7 Days or 31 Days from the View Data menu to determine whether or not there are any trends in the performance of your Grid Control installation.

   Consider adding additional Management Service installations if the response time of all pages is increasing over time or if the response time is unusually high for specific popular pages within the Grid Control Console.

3.5.1.1 Post Management Service - Install Management Repository Configuration

There are some parameters that should be configured during the Management Repository database install (as previously mentioned) and some parameters that should be set after the Management Service has been installed. Once the Enterprise Manager console is available, it can be used to configure these best practices in the Management Repository. These best practices fall in the area of:

• Configuring Storage

• Configuring Oracle Database 10g with RAC for High Availability and Fast Recover Ability

  • Enable ARCHIVELOG Mode

  • Enable Block Checksums

  • Configure the Size of Redo Log Files and Groups Appropriately

  • Use a Flash Recovery Area

  • Enable Flashback Database

  • Use Fast-Start Fault Recovery to Control Instance Recovery Time

  • Enable Database Block Checking

  • Set DISK_ASYNCH_IO

  The details of these settings are available in the Oracle Database High Availability Best Practices.

  Use the MAA Advisor for additional high availability recommendations that should be applied to the Management Repository. To access the MAA Advisor:

  1. On the Database Target Home page, locate the High Availability section.

  2. Click Details next to the Console item.

  3. In the Availability Summary section of the High Availability Console page, click Details next to the MAA Advisor item.

3.5.2.1 Management Service Install Location

If you are managing a large environment with multiple Management Services and Management Repository nodes, then consider installing the Management Services on hardware nodes that are different from Management Repository nodes (Figure 3-4). This allows you to scale out the Management Services in the future.

Figure 3-4 Management Service and Management Repository on Separate Hardware

Also consider the network latency between the Management Service and the Management Repository while determining the Management Service install location. The distance between the Management Service and the Management Repository may be one of the factors that effect network latency and hence determine Enterprise Manager performance.

If the network latency between the Management Service and Management Repository tiers is high or the hardware available for running Enterprise Manager is limited, then the Management Service can be installed on the same hardware as the Management Repository (Figure 3-5). This allows for Enterprise Manager high availability, as well as keep the costs down.

Figure 3-5 Management Service and Management Repository on Same Hardware

3.5.2.2 Configure Management Service to Management Repository Communication

Once all the Management Service processes have been installed, they need to be configured to communicate with each node of the RAC Management Repository in a redundant fashion. To accomplish this, modify the field 'emdRepConnectDescriptor' in the file $ORACLE_HOME/sysman/config/emoms.properities for each installed Management Service. The purpose of this configuration is to make the Management Service aware of all instances in the database cluster that are able to provide access to the Management Repository through the database service 'EMREP'.

Note that Real Application Cluster (RAC) nodes are referred to by their virtual IP (vip) names. The service_name parameter is used instead of the system identifier (SID) in connect_data mode and failover is turned on. Refer to Oracle Database Net Services Administrator's Guide for details.

oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=
(ADDRESS_LIST\=(FAILOVER\=ON) (ADDRESS\=(PROTOCOL\=TCP)(HOST\=node1-vip.example.com)
(PORT\=1521))(ADDRESS\=(PROTOCOL\=TCP)(HOST\=node2-vip.example.com)
(PORT\=1521)) (CONNECT_DATA\=(SERVICE_NAME\=EMREP)))

After making the previous change, run the following command to make the same change to the monitoring configuration used for the Management Services and Repository target: emctl config emrep -conn_desc "<tns alias>"

3.5.2.3 Configure Management Service to Direct Traffic Through SLB

Finally, modify the Management Service to take advantage of the capabilities of the Server Load Balancer. These modifications will cause all the Management Service nodes to redirect Enterprise Manager console traffic through the server load balancer, thereby presenting a single URL to the Enterprise Manager user.

To prevent the browser from bypassing the load balancer when a URL is redirected, Grid Control will now reconfigure the ssl.conf file as a part of resecuring the Management Service. After the Load Balancer is configured, issue the following command from the Management Server home: emctl secure oms

Modify the 'Port' in the Oracle HTTP Server configuration file at $ORACLE_HOME/Apache/Apache/conf/ssl.conf to be '443'. This assumes you are running in the default secured configuration between the Management Service and Management Agent.

3.5.3.1 Configuring Shared File Areas for Management Services

The Management Service for Grid Control 10g Release 2 has a high availability feature called the Shared Filesystem Loader. In the Shared Filesystem Loader, management data files received from Management Agents are stored temporarily on a common shared location called the shared receive directory. All Management Services are configured to use the same storage location for the shared receive directory. The Management Services coordinate internally and distribute amongst themselves the workload of uploading files into the Management Repository. Should a Management Service go down, its workload is taken up by surviving Management Services.

1. Allow all Management Services to process the Management Agent data and take better advantage of available resources.

2. The ability of another Management Service to process Management Agent data in the event of a failure of a Management Service.

   To configure the Management Service to use Shared File system Loader, you must run the following steps:

   1. Stop all Oracle Management Services.

   2. Configure a shared receive directory that is accessible by all Management Services using redundant file system storage.

   3. Execute:

      emctl config oms loader -shared yes -dir <loaderdirectory> individually on all Management Service hosts, where <loaderdirectory> is the full path to the shared receive directory created.

   Note:

   Enterprise Manager will fail to start if all the Management Services are not configured to point to the same shared directory. This shared directory should be on redundant storage.

   Note:

   To modify the location for the deployment procedure library using the Enterprise Manager UI:

   1. Click the Deployments tab on the Enterprise Manager Home page.

   2. Click the Provisioning subtab.

   3. On the Provisioning page, click the Administration subtab.

   4. In the Software Library Configuration section, click Add to set the Software Library Directory Location to a shared storage that can be accessed by any host running the Management Service.

3. Configure Software Library

   Since software library location has to be accessed by all Management Services, considerations similar to shared filesystem loader directory apply here too. Refer to Chapter 16, "Using a Software Library,", for details.

3.5.4.1 Configuring Oracle HTTP Server When Using a Load Balancer for the Grid Control Console

The Management Service is implemented as a J2EE Web application, which is deployed on an instance of Oracle Application Server. Like many Web-based applications, the Management Service often redirects the client browser to a specific set of HTML pages, such as a logon screen and a specific application component or feature.

When the Oracle HTTP Server redirects a URL, it sends the URL, including the Oracle HTTP Server host name, back to the client browser. The browser then uses that URL, which includes the Oracle HTTP Server host name, to reconnect to the Oracle HTTP Server. As a result, the client browser attempts to connect directly to the Management Service host and bypasses the load balancer.

To prevent the browser from bypassing the load balancer when a URL is redirected, Grid Control will now reconfigure the ssl.conf file as a part of resecuring the Management Service. After the Load Balancer is configured, issue the following command from the Management Server home: emctl secure oms

3.5.4.2 Configuring Console URL

Grid Control sends out notifications and reports using e-mail with links pointing back to the Grid Control Console. When a SLB is configured, the e-mails should contain links pointing the SLB and not the individual Management Service. Go to the Management Services and Repository page on Grid Control Console. Click Add Console URL and specify the SLB virtual service used for UI access.

3.5.4.3 Understanding the Flow of Data When Load Balancing the Grid Control Console

Using a load balancer to manage the flow of data from the Management Agents is not the only way in which a load balancer can help you configure a highly available Grid Control environment. You can also use a load balancer to balance the load and to provide a failover solution for the Grid Control Console

Figure 3-6 shows a typical configuration where a load balancer is used between the Management Agents and multiple Management Services, as well as between the Grid Control Console and multiple Management Services.

Figure 3-6 Load Balancing Between the Grid Control Console and the Management Service

Description of "Figure 3-6 Load Balancing Between the Grid Control Console and the Management Service"

In this example, a single load balancer is used for the upload of data from the Management Agents and for the connections between the Grid Control Console and the Management Service.

When you use a load balancer for the Grid Control Console, the management data uses the following paths through the Grid Control architecture:

1. Administrators use one URL to access the Grid Control Console. This URL directs the browser to the load balancer virtual service. The virtual service redirects the browser to one of the Management Service installations. Depending upon the host name and port selected by the load balancer from the virtual pool of Management Service installations, the Grid Control Console obtains the management data by way of OracleAS Web Cache and the Oracle HTTP Server on one of the Management Service hosts.

2. Each Management Agent uploads its data to a common load balancer URL (as described in Section 3.5.5.1) and data is written to the shared receive directory.

3. Each Management Service communicates by way of JDBC with a common Management Repository, just as they do in the multiple Management Service configuration defined in Section 3.4.

4. Each Management Service communicates directly with each Management Agent by way of HTTP, just as they do in the multiple Management Service configuration defined in Section 3.

3.5.5.1 Load Balancing Connections Between the Management Agent and the Management Service

Before you implement a plan to protect the flow of management data from the Management Agents to the Management Service, you should be aware of some key concepts.

Specifically, Management Agents do not maintain a persistent connection to the Management Service. When a Management Agent needs to upload collected monitoring data or an urgent target state change, the Management Agent establishes a connection to the Management Service. If the connection is not possible, such as in the case of a network failure or a host failure, the Management Agent retains the data and reattempts to send the information later.

To protect against the situation where a Management Service is unavailable, you can use a load balancer between the Management Agents and the Management Services.

However, if you decide to implement such a configuration, be sure to understand the flow of data when load balancing the upload of management data.

Figure 3-7 shows a typical scenario where a set of Management Agents upload their data to a load balancer, which redirects the data to one of two Management Service installations.

Figure 3-7 Load Balancing Between the Management Agent and the Management Service

Description of "Figure 3-7 Load Balancing Between the Management Agent and the Management Service"

In this example, only the upload of Management Agent data is routed through the load balancer. The Grid Control Console still connects directly to a single Management Service by way of the unique Management Service upload URL. This abstraction allows Grid Control to present a consistent URL to both Management Agents and Grid Control consoles, regardless of the loss of any one Management Service component.

When you load balance the upload of Management Agent data to multiple Management Service installations, the data is directed along the following paths:

1. Each Management Agent uploads its data to a common load balancer URL. This URL is defined in the emd.properties file for each Management Agent. In other words, the Management Agents connect to a virtual service exposed by the load balancer. The load balancer routes the request to any one of a number of available servers that provide the requested service.

2. Each Management Service, upon receipt of data, stores it temporarily in a local file and acknowledges receipt to the Management Agent. The Management Services then coordinate amongst themselves and one of them loads the data in a background thread in the correct chronological order.

   Also, each Management Service communicates by way of JDBC with a common Management Repository, just as they do in the multiple Management Service configuration defined in Section 3.4.

3. Each Management Service communicates directly with each Management Agent by way of HTTP, just as they do in the multiple Management Service configuration defined in Section 3.4.

3.5.6.1 Prerequisites

The following prerequisites must be satisfied.

• The primary site must be configured as per Grid Control MAA guidelines described in previous sections. This includes Management Services fronted by an SLB and all Management Agents configured to upload to Management Services by the SLB.

• The standby site must be similar to the primary site in terms of hardware and network resources to ensure there is no loss of performance when failover happens.

• There must be sufficient network bandwidth between the primary and standby sites to handle peak redo data generation.

• Configure shared storage used for shared filesystem loader and software library to be replicated at the primary and standby site. In the event of a site outage, the contents of this shared storage must be made available on the standby site using hardware vendor disk level replication technologies.

• For complete redundancy in a disaster recovery environment, a second load balancer must be installed at the standby site. The secondary SLB must be configured in the same fashion as the primary. Some SLB vendors (such as F5 Networks) offer additional services that can be used to pass control of the Virtual IP presented by the SLB on the primary site to the SLB on the standby site in the event of a site outage. This can be used to facilitate automatic switching of Management Agent traffic from the primary site to the standby site.

3.5.6.2 Setup Standby Database

As described earlier, the starting point of this step is to have the primary site configured as per Grid Control MAA guidelines. The following steps will lay down the procedure for setting up the standby Management Repository database.

1. Prepare Standby Management Repository hosts for Data Guard

   Install a Management Agent on each of the standby Management Repository hosts. Configure the Management Agents to upload by the SLB on the primary site.

   Install CRS and Database software on the standby Management Repository hosts. The version used must be the same as that on the primary site.

2. Prepare Primary Management Repository database for Data Guard

   In case the primary Management Repository database is not already configured, enable archive log mode, setup flash recovery area and enable flashback database on the primary Management Repository database.

3. Create Physical Standby Database

   In Enterprise Manager, the standby Management Repository database must be physical standbys. Use the Enterprise Manager Console to setup a physical standby database in the standby environment prepared in previous steps. Note that Enterprise Manager Console does not support creating a standby RAC database. If the standby database has to be RAC, configure the standby database using a single instance and then use the Convert to RAC option from Enterprise Manager Console to convert the single instance standby database to RAC. Also, note that during single instance standby creation, the database files should be created on shared storage to facilitate conversion to RAC later.

4. Add Static Service to Listener

   To enable Data Guard to restart instances during the course of broker operations, a service with a specific name must be statically registered with the local listener of each instance. The value for the GLOBAL_DBNAME attribute must be set to a concatenation of <db_unique_name>_DGMGRL.<db_domain>. For example, in the LISTENER.ORA file:

   SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(SID_NAME=sid_name)
        (GLOBAL_DBNAME=db_unique_name_DGMGRL.db_domain)
        (ORACLE_HOME=oracle_home)))
   

5. Enable Flashback Database on the Standby Database

6. Verify Physical Standby

   Verify the Physical Standby database through the Enterprise Manager Console. Click the Log Switch button on the Data Guard page to switch log and verify that it is received and applied to the standby database.

3.5.6.3 Setup Standby Management Service

The following considerations should be noted before installing the standby Management Services.

• It is recommended that this activity be done during a lean period or during a planned maintenance window. When new Management Services are installed on the standby site, they are initially configured to connect to the Management Repository database on the primary site. Some workload will be taken up by the new Management Service. This could result in temporary loss in performance if the standby site Management Services are located far away from the primary site Management Repository database. However there would be no data loss and the performance would recover once the standby Management Services are shutdown post configuration.

• The shared storage used for the shared filesystem loader and software library must be made available on the standby site using the same paths as the primary site.

Install the Management Service software using steps documented in the Oracle Enterprise Manager Grid Control Installation and Basic Configuration Guide. Refer to the Installing Software-Only and Configuring Later - Additional Management Service option. Specify the primary database connection settings in the response file. Once you have the Management Service installed, use the following steps to copy over the configuration from any the primary Management Service.

1. Export the configuration from the primary Management Service using:

   emctl exportconfig oms -dir <location for the export file>

2. Copy over the exported file to the standby Management Service

3. Shutdown the standby Management Service

4. Import the exported configuration on the standby Management Service using: emctl importconfig oms -file <full path of the export file>

5. Make the standby Management Service point to the standby Management Repository database by updating the emoms.properties file: oracle.sysman.eml.mntr.emdRepConnectDescriptor=<connect descriptor of standby database>

6. Setup EMCLI on the standby Management Service using the URL of the primary SLB using emcli setup -url=https://slb.example.com/em -username sysman -password <sysman password> -nocertvalidate

7. Resecure the Management Agent that is installed with the standby Management Service to upload to primary SLB using emctl secure agent -emdWalletSrcUrl https://slb.example.com:<upload port>/em

8. Update the standby SLB configuration by adding the standby Management Service to the different pools on the SLB. Setup monitors for the new Management Service. Modify the ssl.conf file to set the Port directive to the SLB virtual port used for UI access.

Repeat the previous steps for setting up an additional standby Management Service.

3.5.6.4 Switchover

Switchover is a planned activity where operations are transferred from the Primary site to a Standby site. This is usually done for testing and validation of Disaster Recovery (DR) scenarios and for planned maintenance activities on the primary infrastructure. This section describes the steps to switchover to the standby site. The same procedure is applied to switchover in either direction.

Enterprise Manager Console cannot be used to perform switchover of the Management Repository database. The Data Guard Broker command line tool DGMGRL should be used instead.

1. Prepare Standby Database

   Verify that recovery is up-to-date. Using the Enterprise Manager Console, you can view the value of the ApplyLag column for the standby database in the Standby Databases section of the Data Guard Overview Page.

2. Shutdown the Primary Enterprise Manager Application Tier

   Shutdown all the Management Services in the primary site by running the following command on each Management Service: opmnctl stopall

   Shutdown the Enterprise Manager jobs running in Management Repository database: - alter system set job_queue_processes=0;

3. Verify Shared Loader Directory / Software Library Availability

   Ensure all files from the primary site are available on the standby site.

4. Switchover to the Standby Database

   Use DGMGRL to perform a switchover to the standby database. The command can be run on the primary site or the standby site. The switchover command verifies the states of the primary database and the standby database, affects switchover of roles, restarts the old primary database, and sets it up as the new standby database.

   SWITCHOVER TO <standby database name>;

   Verify the post switchover states:

   SHOW CONFIGURATION;
   SHOW DATABASE <primary database name>;
   SHOW DATABASE <standby database name>;
   

5. Startup the Enterprise Manager Application Tier

   Startup all the Management Services on the standby site: opmnctl startall

   Startup the Enterprise Manager jobs running in Management Repository database on the standby site (the new primary database) - alter system set job_queue_processes=10;

6. Relocate Management Services and Management Repository target

   The Management Services and Management Repository target is monitored by a Management Agent on one of the Management Services on the primary site. To ensure that the target is monitored after switchover/failover, relocate the target to a Management Agent on the standby site by running the following command on one of the Management Service standby sites.

   emctl config emrep -agent <agent name> -conn_desc

7. Switchover to Standby SLB

   Make appropriate network changes to failover your primary SLB to standby SLB that is, all requests should now be served by the standby SLB without requiring any changes on the clients (browser and Management Agents).

This completes the switchover operation. Access and test the application to ensure that the site is fully operational and functionally equivalent to the primary site. Repeat the same procedure to switchover in the other direction.

3.5.6.5 Failover

A standby database can be converted to a primary database when the original primary database fails and there is no possibility of recovering the primary database in a timely manner. This is known as a manual failover. There may or may not be data loss depending upon whether your primary and target standby databases were synchronized at the time of the primary database failure.

This section describes the steps to failover to a standby database, recover the Enterprise Manager application state by resyncing the Management Repository database with all Management Agents, and enabling the original primary database as a standby using flashback database.

The word manual is used here to contrast this type of failover with a fast-start failover described later in Section 3.5.6.6, "Automatic Failover".

1. Verify Shared Loader Directory and Software Library Availability

   Ensure all files from the primary site are available on the standby site.

2. Failover to Standby Database

   Shutdown the database on the primary site. Use DGMGRL to connect to the standby database and execute the FAILOVER command: FAILOVER TO <standby database name>;

   Verify the post failover states:

   SHOW CONFIGURATION;
   SHOW DATABASE <primary database name>;
   SHOW DATABASE <standby database name>;
   

   Note that after the failover completes, the original primary database cannot be used as a standby database of the new primary database unless it is re-enabled.

3. Resync the New Primary Database with Management Agents

   Skip this step if you are running in Data Guard Maximum Protection or Maximum Availability level as there is no data loss on failover. On the other hand, if there is data loss, you need to synchronize the new primary database with all Management Agents.

   On any one Management Service on the standby site, run the following command:

   emctl resync repos -full -name "<name for recovery action>"

   This command submits a resync job that would be executed on each Management Agent when the Management Services on the standby site are brought up.

   Repository resynchronization is a resource intensive operation. A well tuned Management Repository will help significantly to complete the operation as quickly as possible. Refer to Chapter 12, "Sizing and Maximizing the Performance of Oracle Enterprise Manager" for guidelines on routine housekeeping jobs that keep your site running well. Specifically if you are not routinely coalescing the IOTs/indexes associated with Advanced Queueing tables as described in My Oracle Support note 271855.1, running the procedure before resync will significantly help the resync operation to complete faster.

4. Startup the Enterprise Manager Application Tier

   Startup all the Management Services on the standby site by running the following command on each Management Service.

   opmnctl startall

5. Relocate Management Services and Management Repository target

   The Management Services and Management Repository target is monitored by a Management Agent on one of the Management Services on the primary site. To ensure that target is monitored after switchover/failover, relocate the target to a Management Agent on the standby site by running the following command on one of the standby site Management Services.

   emctl config emrep -agent <agent name> -conn_desc

6. Switchover to Standby SLB

   Make appropriate network changes to failover your primary SLB to the standby SLB, that is, all requests should now be served by the standby SLB without requiring any changes on the clients (browser and Management Agents).

7. Establish Original Primary Database as Standby Database Using Flashback

   Once access to the failed site is restored and if you had flashback database enabled, you can reinstate the original primary database as a physical standby of the new primary database.

   • Shutdown all the Management Services in original primary site:

     opmnctl stopall

   • Restart the original primary database in mount state:

     shutdown immediate;

     startup mount;

   • Reinstate the Original Primary Database

     Use DGMGRL to connect to old primary database and execute the REINSTATE command

     REINSTATE DATABASE <old primary database name>;

   • The newly reinstated standby database will begin serving as standby database to the new primary database.

   • Verify the post reinstate states:

     SHOW CONFIGURATION;
     SHOW DATABASE <primary database name>;
     SHOW DATABASE <standby database name>;
     

8. Navigate to the Management Services and Repository Overview page of Grid Control Console. Under Related Links, click Repository Synchronization. This page shows the progress of the resynchronization operation on a per Management Agent basis. Monitor the progress.

   Operations that fail should be resubmitted manually from this page after fixing the error mentioned. Typically, communication related errors are caused by Management Agents being down and can be fixed by resubmitting the operation from this page after restarting the Management Agent.

   For Management Agents that cannot be started due to some reason, for example, old decommissioned Management Agents, the operation should be stopped manually from this page. Resynchronization is deemed complete when all the jobs have a completed or stopped status.

This completes the failover operation. Access and test the application to ensure that the site is fully operational and functionally equivalent to the primary site. Do a switchover procedure if the site operations have to be moved back to the original primary site.

3.5.6.6 Automatic Failover

This section details the steps to achieve complete automation of failure detection and failover procedure by utilizing Fast-Start Failover and Observer process. At a high level the process works like this:

• Fast-Start Failover (FSFO) determines that a failover is necessary and initiates a failover to the standby database automatically

• When the database failover has completed the DB_ROLE_CHANGE database event is fired

• The event causes a trigger to be fired which calls a script that configures and starts Enterprise Manager Application Tier

Perform the following steps:

1. Develop Enterprise Manager Application Tier Configuration and Startup Script

   Develop a script that will automate the Enterprise Manager Application configuration and startup process. See the sample shipped with Grid Control in the OH/sysman/ha directory. A sample script for the standby site is included here and should be customized as needed. Make sure ssh equivalence is setup so that remote shell scripts can be executed without password prompts. Place the script in a location accessible from the standby database host. Place a similar script on the primary site.

   #!/bin/sh
   # Script: /scratch/EMSBY_start.sh
   # Primary Site Hosts
   # Repos: earth, OMS: jupiter1, jupiter2
   # Standby Site Hosts
   # Repos: mars, # OMS: saturn1, saturn2
   LOGFILE="/net/mars/em/failover/em_failover.log"
   OMS_ORACLE_HOME="/scratch/OracleHomes/em/oms10g"
   CENTRAL_AGENT="saturn1.example.com:3872"
    
   #log message
   echo "###############################" >> $LOGFILE
   date >> $LOGFILE
   echo $OMS_ORACLE_HOME >> $LOGFILE
   id >>  $LOGFILE 2>&1
    
   #startup all OMS
   #Add additional lines, one each per OMS in a multiple OMS setup
   ssh orausr@saturn1 "$OMS_ORACLE_HOME/opmn/bin/opmnctl startall" >>  $LOGFILE 2>&1
   ssh orausr@saturn2 "$OMS_ORACLE_HOME/opmn/bin/opmnctl startall" >>  $LOGFILE 2>&1
    
   #relocate Management Services and Repository target
   #to be done only once in a multiple OMS setup
   #allow time for OMS to be fully initialized
   ssh orausr@saturn1 "$OMS_ORACLE_HOME/bin/emctl config emrep -agent $CENTRAL_AGENT -conn_desc -sysman_pwd <password>" >> $LOGFILE 2>&1
    
   #always return 0 so that dbms scheduler job completes successfully
   exit 0
   

2. Automate Execution of Script by Trigger

   Create a database event "DB_ROLE_CHANGE" trigger, which fires after the database role changes from standby to primary. See the sample shipped with Grid Control in OH/sysman/ha directory.

   --
   --
   -- Sample database role change trigger
   --
   --
   CREATE OR REPLACE TRIGGER FAILOVER_EM
   AFTER DB_ROLE_CHANGE ON DATABASE
   DECLARE
       v_db_unique_name varchar2(30);
       v_db_role varchar2(30);
   BEGIN
       select upper(VALUE) into v_db_unique_name
       from v$parameter where NAME='db_unique_name';
       select database_role into v_db_role
       from v$database;
    
       if v_db_role = 'PRIMARY' then
    
         -- Submit job to Resync agents with repository
         -- Needed if running in maximum performance mode
         -- and there are chances of data-loss on failover
         -- Uncomment block below if required
         -- begin
         --  SYSMAN.setemusercontext('SYSMAN', SYSMAN.MGMT_USER.OP_SET_IDENTIFIER);
         --  SYSMAN.emd_maintenance.full_repository_resync(('AUTO-FAILOVER to '||v_db_unique_name);
         --  SYSMAN.setemusercontext('SYSMAN', SYSMAN.MGMT_USER.OP_CLEAR_IDENTIFIER);
         -- end;
    
         -- Start the EM mid-tier
         dbms_scheduler.create_job(
             job_name=>'START_EM',
             job_type=>'executable',
             job_action=> '<location>' || v_db_unique_name|| '_start_oms.sh',
             enabled=>TRUE
         );
       end if;
   EXCEPTION
   WHEN OTHERS
   THEN
       SYSMAN.mgmt_log.log_error('LOGGING', SYSMAN.MGMT_GLOBAL.UNEXPECTED_ERR,
   SYSMAN.MGMT_GLOBAL.UNEXPECTED_ERR_M || 'EM_FAILOVER: ' ||SQLERRM);
   END;
   /
    
   

   Note:

   Based on your deployment, you might require additional steps to synchronize and automate the failover of SLB and shared storage used for loader receive directory and software library. These steps are vendor specific and beyond the scope of this document. One possibility is to invoke these steps from the Enterprise Manager Application Tier startup and configuration script.

3. Configure Fast-Start Failover and Observer

   Use the Fast-Start Failover configuration wizard in Enterprise Manager Console to enable FSFO and configure the Observer.

   This completes the setup of automatic failover.

3.7.4.1 Upload Delay for Monitoring Data

When assessing the health and availability of targets through the Grid Control console, information is slow to appear in the UI, especially after a Management Service outage. The state of a target in the Grid Control console may be delayed after a state change on the monitored host. Use the Management System page to gauge backlog for pending files to be processed.

3.7.4.2 Notification Delay of Target State Change

The model used by the Management Agent to assess the state of health for any particular monitored target is poll based. Management Agents immediately post a notification to the Management Service as soon as a change in state is detected. This infers that there is some potential delay for the Management Agent to actually detect a change in state.