2 Oracle Application Server High Availability Framework

Whereas Chapter 1 provided an overview of high availability in general, this chapter describes the Oracle Application Server features that are important in high availability topologies. It contains the following sections:

Section 2.1, "Process Management through OPMN"
Section 2.2, "Replication of State Information"
Section 2.3, "Load Balancing in Oracle Application Server"
Section 2.4, "OracleAS Clusters"
Section 2.5, "External Load Balancers"
Section 2.6, "Backup and Recovery"
Section 2.7, "Disaster Recovery"

Section 2.8, "High Availability Topologies: Overview"

2.1 Process Management through OPMN

An Oracle Application Server instance consists of many different running processes that serve client requests. Ensuring high availability means ensuring that all these processes run smoothly, fulfill requests, and do not experience any unexpected hangs or failures.

The Oracle Process Manager and Notification Server (OPMN) component of Oracle Application Server provides the following process management services:

When OPMN detects Oracle Application Server processes that are down, unresponsive, or unreachable, it automatically restarts them. It monitors the processes by pinging them or through notification methods.
OPMN starts processes in the proper order: processes that depend on other processes are not started up until the dependent processes are started.

You can also use OPMN to perform tasks such as starting and stopping processes, and also to check the status of processes. See the Oracle Process Manager and Notification Server Administrator's Guide for details.

OPMN manages the following Oracle Application Server processes:

Oracle HTTP Server
Oracle Containers for J2EE (OC4J)
OracleAS Log Loader
OracleAS Guard (for disaster recovery)
OracleAS Port Tunnel

In addition, OPMN implicitly manages any applications that rely on the above components. For example, J2EE applications are managed by OPMN because they run under OC4J, and OC4J is managed by OPMN.

You also use OPMN to cluster Oracle Application Server instances so that Oracle HTTP Server can direct requests to any OC4J instance in a cluster. This clustering is needed in active-active topologies. See Chapter 3, "Active-ActiveTopologies" for details.

OracleAS Clusters also enable you to manage all the Oracle Application Server instances in the cluster. For example, you can issue an OPMN command on one machine to start all processes or a specific process type across all local and remote Oracle Application Server instances in the cluster.

OPMN is extensible, providing the capability to add information about custom processes including load environment information, stopping procedures, and methods for death detection and restart.

For details on OPMN, see the Oracle Process Manager and Notification Server Administrator's Guide.

2.2 Replication of State Information

One of the advantages of distributing applications is that multiple redundant processes can all handle requests from clients. In the event that one of these processes becomes unavailable, another process can service the request.

Some applications may require Oracle Application Server to maintain stateful information across consecutive requests. In order to provide transparent failover of these requests, it is necessary to replicate this application state across multiple processes. Oracle Application Server enables the replication of state in J2EE applications through application-level clustering. In an application cluster, several processes work together to deliver the same J2EE application and replicate the state created by it. This enables the transparent failover of requests between the instances in the cluster.

A J2EE application can maintain two types of state information:

HTTP session state (updated by servlets and JSPs)
Stateful session EJB state (updated by stateful session EJB instances)

Within an OracleAS Cluster (OC4J), application-level clustering enables the replication of both types of state information. You can control how the state information is replicated, the frequency at which the state information is replicated between OC4J instances, and which state information is replicated. See Table 2-1:

Table 2-1 Procedures for Replicating State Information

To do this:	See
Set up how state information is replicated	You can replicate state information using multicasting, peer-to-peer, or database: Section 3.2.2, "Setting up Multicast Replication" Section 3.2.3, "Setting up Peer-to-Peer Replication" Section 3.2.4, "Setting up Replication to a Database"
Specify how frequently state information is replicated, or specify which information is replicated	See Section 3.2.5, "Setting the Replication Policy".

To do this:

See

Set up how state information is replicated

You can replicate state information using multicasting, peer-to-peer, or database:

Specify how frequently state information is replicated, or specify which information is replicated

See Section 3.2.5, "Setting the Replication Policy".

See also the "Application Clustering in OC4J" chapter in the Oracle Containers for J2EE Configuration and Administration Guide for more information.

2.3 Load Balancing in Oracle Application Server

Load balancing involves distributing requests among two or more processes. Features of a software or hardware load balancer include:

load balancing algorithm

The algorithm specifies how to allocate requests across the different processes. The most common load balancing algorithms include simple round-robin or assignment based on some weighted property of the instance such as the response time or capacity of that instance relative to other instances.

The algorithms usually include processes in the local instance as well as in remote instances. This enables one component to direct requests to another component running on a remote machine.
death detection

The load balancer must be able to recognize failed requests to one or more processes and be able to mark these processes as inactive so that no further requests will be forwarded to them.

In Oracle Application Server, routing of requests between some components involve load balancing mechanisms. Table 2-2 shows some examples:

Table 2-2 Routing of Requests Between Components

From	To	Description
OracleAS Web Cache (from Release 2 (10.1.2))	Oracle HTTP Server	OracleAS Web Cache can route requests to any Oracle Application Server instance that includes Oracle HTTP Server. If OracleAS Web Cache detects failures in the replies returned by Oracle HTTP Server, it routes new requests to the available Oracle HTTP Servers. You configure the routing algorithm in the OracleAS Web Cache component. See the Oracle Application Server Web Cache Administrator's Guide from Release 2 (10.1.2) for details.
Oracle HTTP Server	OC4J	Oracle HTTP Server routes requests for J2EE applications to OC4J processes. The mod_oc4j module in Oracle HTTP Server performs the routing. mod_oc4j can route a request to any OC4J process in the OracleAS Clusters. Oracle HTTP Server maintains a routing table of available OC4J processes and routes new requests only to those OC4J processes that are up and running. You configure the routing algorithm using directives in Oracle HTTP Server configuration files. See Section 3.2.12, "Setting mod_oc4j Load Balancing Options" for details.
Oracle HTTP Server	Database	Oracle HTTP Server routes requests for PL/SQL applications to the database. The mod_plsql module in Oracle HTTP Server performs the routing. mod_plsql is able to detect failure in the database and it routes requests only to available database nodes.
OC4J	OC4J	Within OC4J itself, OC4J routes requests from the presentation layer components (servlets and JSPs) to the business layer components (EJBs). If OC4J detects failures in the RMI invocations to the EJB tier, it fails over communication to available EJB nodes.
OC4J	Database	OC4J drivers are enabled to detect failures of database nodes and re-route requests to available nodes.

2.4 OracleAS Clusters

Note that Oracle Application Server supports different types of clustering. This section describes OracleAS Clusters. For information on application-level clustering, see Section 2.2, "Replication of State Information".

You can group Oracle Application Server instances in OracleAS Clusters. OracleAS Clusters provide the following benefits:

In active-active topologies, you need to group Oracle Application Server instances in the same cluster so that Oracle HTTP Server instances in the topology know which OC4J instances are also in the same topology. These are the OC4J instances that Oracle HTTP Server can forward application requests to.

When Oracle HTTP Server applies the load balancing algorithm specified in the mod_oc4j module, it applies the algorithm to the OC4J instances in the cluster. To set the mod_oc4j load balancing algorithm, see Section 3.2.12, "Setting mod_oc4j Load Balancing Options".
You can manage the Oracle Application Server instances collectively by using the @cluster parameter in the opmnctl command. For example, you can stop Oracle HTTP Server in all the instances belonging to the cluster with the following command:
```
> opmnctl @cluster stopproc ias-component=HTTP_Server
```
For a list of opmnctl commands that accept the @cluster parameter, see the Oracle Process Manager and Notification Server Administrator's Guide.
If you are using application-level clustering with dynamic peer-to-peer replication, you need to set up OracleAS Clusters because OracleAS Clusters determine the members of the cluster.

For information on peer-to-peer replication, see Section 3.2.3, "Setting up Peer-to-Peer Replication".

2.5 External Load Balancers

To load balance requests among many Oracle Application Server instances in an active-active topology, you should use an external load balancer.

When several Oracle Application Server instances are clustered and are fronted by a load balancer, the load balancer hides the multiple instance configuration by being the entry point to the system. External load balancers can send requests to any Oracle Application Server instance in the cluster, as any instance can service any request. You can raise the capacity of the system by introducing additional Oracle Application Server instances. These instances can be installed on separate nodes to allow for redundancy in case of node failure.

There are different types of external load balancers you can use with Oracle Application Server. Table 2-3 summarizes the different types.

Table 2-3 Types of External Load Balancers

Load Balancer Type	Description
Hardware load balancer	Hardware load balancing involves placing a hardware load balancer in front of a group of Oracle Application Server instances or OracleAS Web Cache (from Release 2 (10.1.2)). The hardware load balancer routes requests to the Oracle HTTP Server or OracleAS Web Cache instances in a client-transparent fashion.
Software load balancer	Software load balancer involves using some process that intercepts the different calls to an application server and routes those requests to redundant components.
Lvs network load balancer for Linux	With some Linux operating systems, you can use the operating system to perform network load balancing.
Windows Network Load Balancer (applicable to Windows version of Oracle Application Server)	With some Windows operating systems, you can use the operating system to perform network load balancing. For example, with Microsoft Advanced Server, the NLB functionality enables you to send requests to different machines that share the same virtual IP or MAC address. The servers themselves to do not need to be clustered at the operating system level.

External Load Balancer Requirements

Oracle does not provide external load balancers. You can get external load balancers from other companies.

To ensure that your external load balancer can work with Oracle Application Server, check that your external load balancer meets the requirements listed in Table 2-4.

Note that you may not need all the requirements listed in the table. The requirements for an external load balancer depend on the topology being considered, and on the Oracle Application Server components that are being load balanced.

Table 2-4 External Load Balancer Requirements

External Load Balancer Requirement	Description
Virtual servers and port configuration	You need to be able to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements: The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for OracleAS Clusters, the load balancer needs to be configured with a virtual server and ports for HTTP and HTTPS traffic. The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.
Resource monitoring / port monitoring / process failure detection	You need to set up the external load balancer to detect service and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.
Fault tolerant mode	It is highly recommended that you configure the load balancer to be in fault-tolerant mode.
Other	It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the backend services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client machine.

Figure 2-1 depicts an example deployment of a hardware load balancing router with Oracle Application Server.

Figure 2-1 Example load balancing router deployment with Oracle Application Server

Description of "Figure 2-1 Example load balancing router deployment with Oracle Application Server"

Load balancing improves scalability by providing an access point through which requests are routed to one of many available instances. Instances can be added to the group that the external load balancer serves to accommodate additional users.Load balancing improves availability by routing requests to the most available instances. If one instance goes down, or is particularly busy, the external load balancer can send requests to another active instance.

2.6 Backup and Recovery

Protecting against data loss in any system component is critical to maintaining a highly available environment. You should perform regular backups of all Oracle Application Server instances in your environment.

A complete Oracle Application Server environment backup includes:

A full backup of all files in the middle-tier Oracle homes (this includes Oracle software files and configuration files).
A full backup of the Oracle system files on each host in your environment.
If you are using Oracle Identity Management from Release 2 (10.1.2), then you also need to perform a full backup of:
- all files in the Oracle Identity Management's Oracle home (this includes Oracle software files and configuration files)
- the OracleAS Metadata Repository used by the Oracle Identity Management

2.6.1 Oracle Application Server Backup and Recovery Tool

The most frequently changing critical files in an Oracle installation are configuration files and data files. Oracle Application Server provides the OracleAS Backup and Recovery Tool to back up these files.

You can use the OracleAS Backup and Recovery Tool to back up and recover the following installation types:

J2EE server
Oracle HTTP Server
Integrated (J2EE server and Oracle HTTP Server)

The OracleAS Backup and Recovery Tool is installed by default when you install Oracle Application Server. It is installed in the ORACLE_HOME/backup_restore directory.

For details on the OracleAS Backup and Recovery Tool, see the Oracle Application Server Administrator's Guide.

2.7 Disaster Recovery

Disaster recovery refers to how a system can be recovered from catastrophic site failures caused by natural or unnatural disasters. Additionally, disaster recovery can also refer to how a system is managed for planned outages. For most disaster recovery situations, the solution involves replicating an entire site, not just pieces of hardware or subcomponents. This also applies to the OracleAS Disaster Recovery solution.

In the most common configuration, a standby site is created to mirror the production site. Under normal operation, the production site actively services client requests. The standby site is maintained to mirror the applications and content hosted by the production site.

2.7.1 Oracle Application Server Guard

OracleAS Guard automates the restoration of a production site on its corresponding standby site. To protect a complete Oracle Application Server environment from disasters, OracleAS Guard performs the following operations:

Instantiates the standby site: instantiates an Oracle Application Server standby farm that mirrors a primary farm.
Verifies configuration: verifies that a farm meets the requirements to be used as a standby farm for the corresponding primary farm.
Site synchronization: synchronizes the production and the standby sites.

2.8 High Availability Topologies: Overview

Oracle Application Server provides redundancy by supporting multiple instances for the same workload. These redundant configurations provide increased availability through a distributed workload, a failover setup, or both.

From the entry point to an Oracle Application Server system (content cache) to the back end layer (data sources), all the tiers that are crossed by a request can be configured in a redundant manner with Oracle Application Server. The configuration can be an active-active configuration using OracleAS Clusters or an active-passive configuration using OracleAS Cold Failover Cluster.

The following sections describe the basics of these configurations:

Section 2.8.1, "Active-Active Topologies"
Section 2.8.2, "Active-Passive Topologies: OracleAS Cold Failover Clusters"

2.8.1 Active-Active Topologies

Oracle Application Server provides an active-active redundant model for all its components. In an active-active topology, two or more Oracle Application Server instances are configured to serve the same workload. These instances can run on the same machine or on different machines.The instances are front-ended by an external load balancer, which directs requests to any of the active instances. Instead of an external load balancer, you can also run a software load balancer to distribute the requests. In production environment, however, a hardware load balancer is recommended.

Common properties of an active-active topology include:

Similar instance configuration

The instances need to serve the same workload or applications. Some configuration properties should have similar values across instances so that the instances can deliver the same reply to the same request. Other configuration properties may be instance-specific, such as local host name information.

If you make a configuration change to one instance, you should also make the same change to the other instances in the active-active topology. The "Configuring and Managing Clusters" chapter in the Oracle Containers for J2EE Configuration and Administration Guide lists the files that contain properties that should be replicated.
Independent operation

If one Oracle Application Server instance in an active-active topology fails, the other instances in the cluster continue to serve requests. The load balancer directs requests only to instances that are alive.

Advantages of an active-active topology include:

Increased availability

An active-active topology is a redundant configuration. Loss of one instance can be tolerated because other instance can continue to serve the same requests.
Increased scalability and performance

Multiple identically-configured instances provide the capability to share a workload among different machines and processes. You can scale the topology by adding new instances as the number of requests increase.

2.8.2 Active-Passive Topologies: OracleAS Cold Failover Clusters

Oracle Application Server provides an active-passive model for all its components in OracleAS Cold Failover Clusters. In an OracleAS Cold Failover Cluster topology, two Oracle Application Server instances are configured to serve the same application workload but only one is active at any particular time. The passive instance runs (that is, becomes active) only when the active instance fails. These instances run on nodes that are in a hardware cluster.

Common properties of an OracleAS Cold Failover Cluster topology include:

Hardware cluster

In an OracleAS Cold Failover Cluster topology, you run Oracle Application Server on machines that are in a hardware cluster, with vendor clusterware running on the machines.
Shared storage

You install the Oracle home for the Oracle Application Server instance on storage shared by the machines in the hardware cluster.

The active node in the OracleAS Cold Failover Cluster topology mounts the shared storage so that it has access to the Oracle home. If it fails, the passive instance mounts the shared storage and accesses the same Oracle home.
Virtual hostname

The virtual hostname gives clients a single system view of the Oracle Application Server middle tier. Clients use the virtual hostname to access the Oracle Application Server middle tier.

The virtual hostname is associated with a virtual IP. This name-IP entry must be added to the DNS that the site uses. For example, if the two physical hostnames of the hardware cluster are node1.mycompany.com and node2.mycompany.com, the single view of this cluster can be provided by the virtual hostname apps.mycompany.com. In the DNS, apps maps to a virtual IP address that floats between node1 and node2 via a hardware cluster. Clients access Oracle Application Server using apps.mycompany.com; they do not know which physical node is active and actually servicing a particular request.

You can specify the virtual hostname during installation. See the Oracle Application Server Installation Guide.
Failover procedure

An active-passive configuration also includes a set of scripts and procedures to detect failure of the active instance and fail over to the passive instance while minimizing downtime.

Advantages of an OracleAS Cold Failover Cluster topology include:

Increased availability

If the active instance fails for any reason or must be taken offline, an identically configured passive instance is prepared to take over at any time.
Reduced operating costs

In an active-passive topology only one set of processes is up and serving requests. Managing the active instance is generally easier than managing an array of active instances.
Application independence

Some applications may not be suited to an active-active topology. This may include applications that rely heavily on application state or on information stored locally. An active-passive topology has only one instance serving requests at any particular time.