Skip Headers

Oracle® Application Server Performance Guide 10g Release 3 (10.1.3) B25213-01 |
| Home | Solution Area | Contents | Index

Previous | Next |

# 3 Top Performance Areas

This chapter provides a description of top tuning areas for Oracle Application Server and includes the following sections:

- Top Performance Areas

- Advanced Performance Areas

## 3.1 Top Performance Areas

This section covers critical Oracle Application Server performance issues and provides a quickstart for tuning J2EE applications that run on OC4J. Table 3-1 lists a quick guide for performance considerations for Oracle Application Server.

Table 3-1 Top Performance Areas for Oracle Application Server Applications

Performance Area | Description and Reference
--- | ---
Ensure Sufficient Hardware Resources | Oracle Application Server has minimum hardware requirements that enable you to use the product. You also need to run on hardware that supports your application's needs, including the database tier. For details on Oracle Application Server installation requirements, see the Requirements chapter in the Installation Guide your platform. See "Ensure Sufficient Hardware Resources" for more information on platform specific tools that can help you determine if your hardware resources are sufficient.
Ensure Sufficient Java Heap Size | To improve the performance of a J2EE application, you need to provide an adequate heap size for the JVM where the application runs. If the OC4J running your J2EE application does not have enough memory, performance can suffer due to the overhead required to manage limited memory. See "Ensure Sufficient Java Heap for OC4J" for details on setting JVM heap size options.
Tune the JVM for Garbage Collection | To improve the performance of a J2EE application and ensure that your application performance is not being negatively impacted by JVM garbage collection, you need to manage the heap size and sometimes you also need to set JVM options that impact garbage collection frequency. See "Tune the JVM Garbage Collection Options" for more information on garbage collection.
Reuse Database Connections | By default, OC4J data sources enable database connection pooling. Thus, OC4J manages database connections to avoid frequently reestablishing new connections. The Oracle Application Server data source facility provides options you can use to control the number of database connections maintained, and how long they are maintained. See "Reuse Database Connections".
Specify Sufficient HTTP Connections | Tune the Oracle HTTP Server directives to set the level of concurrency by specifying the number of HTTP connections. See "Specify Sufficient Oracle HTTP Server Connections".
Enable JDBC Statement Caching Option | By enabling statement caching to lower the overhead of repeated cursor creation and repeated statement parsing and creation, you can improve performance for applications using the database tier. See "Enable Statement Caching for Data Sources".
Ensure the Database is Properly Tuned | For applications that access a database, ensure that your database is properly configured to support your application's requirements. See "Verify Database Tuning".
Verify Logging Levels | You need to make sure that the logging level is not set higher than the default INFO level logging. If the logging setting does not match the default level, reset the logging level to the default for best performance. See "Verify Logging Levels".
Reuse EJB Instances | Set the EJB options for creating and reusing instances to improve EJB performance. See "Reuse EJB Instances".

### 3.1.1 Ensure Sufficient Hardware Resources

A most crucial performance area is ensuring that there are sufficient CPU, memory and network resources to support the user population and application requirements for your Oracle Application Server installation. You need to monitor resource utilization over an extended period to determine if you have occasional peaks of usage or whether a resource is consistently saturated. You also need to define the acceptable response times and throughputs for applications running at your site, for both peak and extended periods. Also, check the system while running your application under normal load and monitor operating system statistics, including, CPU, memory, disk, and network performance to determine if any hardware resource is saturated.

To check the CPU, memory, and disk performance you can use the following commands:

On Linux systems use the `sar` or `mpstat` command.

On Windows systems use the `perfmon` command.

To check network performance, you can use the following commands:

On Linux and Windows systems:

    % netstat

On Windows systems, you can also use the Windows Task Manager to check network performance.

If any of the hardware resources are saturated, this could be due to one or more of the following:

- The hardware resources are insufficient to run the application.

- The system is not properly configured.

- The application or database needs to be tuned.

For a consistently saturated resource, the solutions are to reduce load or increase resources. For peak traffic periods, if the increased response time is not acceptable the alternatives are to again increase resources or to determine if there is traffic that can be rescheduled to reduce the peak load, such as scheduling batch or background operations during slower periods. Oracle Application Server provides a variety of mechanisms to help you control resource concurrency; this can limit the impact of bursts of traffic. However, for a consistently saturated system, these mechanisms should be viewed as temporary solutions.

See Also: "Specify Sufficient Oracle HTTP Server Connections" "Managing Concurrency and Limiting Connections"

### 3.1.2 Ensure Sufficient Java Heap for OC4J

If you have sufficient memory available on your system and your application is memory intensive, increase the JVM heap size from the default value. While the amount of heap required varies based on the application and on the available memory, for most OC4J server applications, if you have sufficient memory, then Oracle recommends using a heap size of 512 Megabytes or larger.

You can improve performance by setting the minimum heap size equal to the maximum heap size. To do this set the option `-Xms` size to be the same as the `-Xmx` size. To change the size of the heap allocated to the OC4J process in an OC4J instance, specify the following JVM options:

    -Xmssizem -Xmxsizem

If your Oracle Application Server topology includes more than one JVM on the same system, then to maximize performance, set the maximum heap size to accommodate application requirements and make sure that the total memory consumed by all of the JVMs running on the system does not exceed the memory capacity of your system.

When you are running an OC4J instance in an Oracle Application Server managed environment, set the Java heap size by including the `-Xms` and `-Xmx` arguments in the `start-parameters` `java-options` element in the `opmn.xml` file.

For example, the following `opmn.xml` file includes these arguments:

    <ias-component id="OC4J">  <process-type id="home" module-id="OC4J" status="enabled">     <module-data>
            <category id="start-parameters">
               <data id="java-options" value="-Xms512m -Xmx512m . . ."/>
               ...
            </category>
            ...
         </module-data>
         ...
       </process-type>
       ...
    </ias-component>

See Also: The JVM Metrics page in Application Server Control Console. This page is available from the OC4J home page, by clicking the Performance Secondary tab, and then, in the Related Links area, clicking JVM Metrics. You can find detailed information about JVM options and their impact on performance on the JVM vendor's Web sites, such as `http://java.sun.com/performance/reference/whitepapers/5.0_performance.html`

### 3.1.3 Tune the JVM Garbage Collection Options

JVM garbage collection is an expensive operation that can have an impact on application performance; inefficient garbage collection can severely degrade application performance. Therefore, it is important to understand how applications create and destroy objects.

To tune the JVM garbage collection options you need to analyze garbage collections data and check for the frequency and type of garbage collections, the size of the memory pools, and the time spent on garbage collection.

In order to determine application memory requirements you can monitor JVM garbage collection and memory pool sizes using the following:

- The JVM options:

      -verbose:gc
      -XX:+PrintGCDetails

  Look for "Full GC" to identify major collections.

- `jstat` tool

- `visualgc` tool

- The Application Server Control Console JVM Metrics page shows JVM memory pool and garbage collector information. This page is available from the OC4J home page by clicking the Performance Secondary tab, and then, in the Related Links area, clicking JVM Metrics.

Set the -`XX:+AggressiveHeap` JVM option to tune internal VM parameters, and increase the total heap size, as described in "Ensure Sufficient Java Heap for OC4J" to reduce the overhead associated with Full CG garbage collections. The -`XX:+AggressiveHeap` option tunes internal VM parameters to be optimal for long-running, memory-intensive workloads. This option should follow, on the command line, the heap sizing options `-Xms` and `-Xmx`. See "Ensure Sufficient Java Heap for OC4J" for details on setting Java command line options in an Oracle Application Server managed environment (please disregard older versions of Sun documentation which advise against using `-XX:+AggressiveHeap`).

Note: The JVM provides a variety of parameters to allow you to more finely tune heap management and garbage collection behavior. See the following links for a detailed description of these topics.

To determine if an application uses explicit garbage collection, which can have a negative impact on performance, set the `-XX:+DisableExplicitGC` option. This debugging option disables explicit garbage collection. Applications should avoid the use of `system.gc()` calls. If you suspect an application may be explicitly triggering garbage collection, set this parameter and observe the differences in your garbage collection behavior. If you determine that the application is making explicit `system.gc()` calls, discuss with the application developer why this was done and the impact of disabling the calls. Application developers sometimes use `system.gc()` calls to trigger finalizers. This is not a recommended practice and can yield indeterminate behavior .

See Also: `http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf` `http://java.sun.com/docs/hotspot/gc5.0/ergo5.html` `http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html`

### 3.1.4 Reuse Database Connections

To obtain better performance in your application, by lowering the overhead of creating and re-creating database connections, specify the connection pool `min-connections` attribute to set the minimum number of connections that the connection pool maintains.

By default, the value of `min-connections` is 0. For best performance, you should specify a value for `min-connections` other than 0. If `min-connections` is set to a value other than zero, the specified number of connections is maintained; OC4J maintains the connections when they are not in use and they do not time out when the specified `inactivity-timeout` is reached. The `min-connections` attribute does not specify that OC4J pre-create connections at startup. Specify the `initial-limit` attribute to set the number of connections in the connection pool when the pool is initially created or reinitialized. Oracle recommends that you set the `initial-limit` attribute to the same value as the `min-connections` attribute.

If the specified value for `min-connections` is less than `max-connections`, then you should set the `inactivity-timeout` to make sure that connections only time out after an appropriately long period of inactivity. The connection pool `inactivity-timeout` specifies the time, in seconds, to cache unused connections before closing the connection.

To improve performance, you can set the `inactivity-timeout` to a value that allows the connection pool to avoid dropping and then re-acquiring connections while your J2EE application is running. The default value for the `inactivity-timeout` is 60 seconds, which is typically too low for frequently accessed applications where there may be some inactivity between requests. For most applications, to improve performance, Oracle recommends that you increase the `inactivity-timeout` to 120 seconds.

To determine if the default `inactivity-timeout` is too low, monitor your system. If you see that the number of database connections grows and then shrinks during an idle period, and grows again soon after that, you have two options: you can increase the `inactivity-timeout`, or you can increase the `min-connections`.

Notes for reusing database connections:

1. Limiting the total number of open database connections to a number your database can handle is an important tuning consideration. You should check with your database administrator to make sure that the database is configured to support a number of connections that is greater than the following:

   At least as large a number of connections as the sum of the values specified for all the connection pool `min-connections` that could be concurrently active, and as large as the maximum desired concurrency across all the datasources for the database.

2. If the `min-connections` is set to a value other than zero, the specified number of connections is maintained; OC4J maintains the connections when they are not in use, and they do not time out when the specified `inactivity-timeout` is reached.

   Once the specified connections are opened, you need to either stop OC4J or use the refresh operation to close the connections. Application Server Control shows the refresh operation in Connection Pool area on the JDBC Resources page. Click the icon in the Refresh Connection Pool field to initiate a refresh operation.

### 3.1.5 Specify Sufficient Oracle HTTP Server Connections

The Oracle HTTP Server `MaxClients` directive limits the number of clients that can simultaneously connect to your web server, and thus the number of httpd processes.

For Windows, the analogous parameter is `ThreadsPerChild`. The `Oc4jCacheSize` directive specifies the maximum number of idle connections that `mod_oc4j` maintains per OC4J JVM, relevant only on Windows.

You can use the `MaxClients`, `ThreadsPerChild`, and `Oc4jCacheSize` directives to limit incoming connections to the OC4J instances from the Oracle HTTP Server. This section covers the following topics:

- Configuring the MaxClients Directive (for UNIX)

- Configuring the ThreadsPerChild Directive (for Windows)

- Configuring the Oc4jCacheSize Directive

Note: This discussion in this section only applies for the default Oracle HTTP Server supplied with Oracle Application Server (based on Apache 1.3). This discussion does not apply for the Apache 2.0 based standalone version of Oracle HTTP Server.

#### 3.1.5.1 Configuring the MaxClients Directive (for UNIX)

You can configure the `MaxClients` directive in the `httpd.conf` file up to a maximum of 8K (the default value is 150). If your system is not resource-saturated and you have a user population of more than 150 concurrent http connections, you can improve your performance by increasing `MaxClients` to increase server concurrency. Increase `Maxclients` until your system becomes fully utilized (85% is a good threshold).

When system resources are saturated, increasing `MaxClients` does not improve performance. In this case, the `MaxClients` value could be reduced as a throttle on the number of concurrent requests on the server.

If the server handles persistent connections, then it may require sufficient concurrent httpd server processes to handle both active and idle connections. When you specify `MaxClients` to act as a throttle for system concurrency, you need to consider that persistent idle httpd connections also consume httpd processes. Specifically, the number of connections includes the currently active persistent and non-persistent connections and the idle persistent connections. A persistent, `KeepAlive`, http connection consumes an httpd child process, or thread, for the duration of the connection, even if no requests are currently being processed for the connection.

If you have sufficient capacity, `KeepAlive` should be enabled; using persistent connections improves performance and prevents wasting CPU resources reestablishing HTTP connections. Normally, you should not need to change `KeepAlive` parameters.

Note: The default maximum requests for a persistent connection is 100, as specified with the `MaxKeepAliveRequests` directive in `httpd.conf`. By default, the server waits for 15 seconds between requests from a client before closing a connection, as specified with the `KeepAliveTimeout` directive in `httpd.conf`.

When there are no httpd processes available, connection requests are queued in the TCP/IP system until a process becomes available, and eventually clients terminate connections.

#### 3.1.5.2 Configuring the ThreadsPerChild Directive (for Windows)

You can configure the `ThreadsPerChild` directive in the `httpd.conf` file up to a maximum of 8K (the default value is 50). The `ThreadsPerChild` parameter on Windows systems works like the `MaxClients` parameter on UNIX systems.

See Also: "Configuring the MaxClients Directive (for UNIX)"

#### 3.1.5.3 Configuring the Oc4jCacheSize Directive

The `Oc4jCacheSize` directive specifies the maximum number of idle connections that `mod_oc4j` maintains per OC4J JVM. On Windows only, it is sometimes useful to change the default value of this directive.

On UNIX systems where each Oracle HTTP Server process is single threaded, the only meaningful values are 1 which is the default value, and zero (0). A value of zero (0) specifies that Oracle HTTP Server should not maintain any connections and should open a new connection for every request. Since each process is single threaded, a process never needs more than one connection and hence a value of 1 or greater has the same effect on UNIX systems. For best performance, on UNIX systems, do not change the default value for `Oc4jCacheSize`.

On Windows systems, the default Oc4jCacheSize value is 75% of the value of `ThreadsPerChild`; the connection cache is shared among threads in the child process. If Oracle HTTP Server is serving a mixed load of static content along with OC4J requests, then the default should be adequate. If the user's load is all OC4J requests, that is, Oracle HTTP Server serves up little or no content and serves just as a front end for OC4J, then it is a good idea to set `Oc4jCacheSize` equal to `ThreadsPerChild`. This setting provides a dedicated connection per thread, if needed, and should give the best performance.

### 3.1.6 Enable Statement Caching for Data Sources

Enable statement caching to lower the overhead of repeated cursor creation and repeated statement parsing and creation by setting the `num-cached-statements` attribute to a value greater than 0 (the default value is 0, disabled). The number you set for `num-cached-statements` should be the number of SQL statements that you use in your application.

### 3.1.7 Verify Database Tuning

To achieve optimal performance in Oracle Application Server, for applications that use the database, the database tables you access need to be designed with performance in mind and you need to monitor and tune the database server to assure that the system is performant.

This section covers the following:

- Tuning init.ora Database Parameters

- Tuning Redo Logs Location and Sizing

- Automatic Segment-Space Management (ASSM)

See Also: Oracle Database Performance Tuning Guide

#### 3.1.7.1 Tuning init.ora Database Parameters

Table 3-2 shows tuning information for several of the init.ora database initialization parameters.

Table 3-2 Important init.ora Database Tuning Parameters

nit.ora Parameter | Description
--- | ---
`DB_BLOCK_SIZE` | The default block size of 8K is optimal for most systems. However, OLTP systems occasionally benefit from smaller block sizes, and DSS systems occasionally benefit from larger block sizes. See Also: table 8-3, "Block Size Advantages and Disadvantages" in the Oracle Database Performance Tuning Guide.
`PGA_AGGREGATE_TARGET` | Specifies the target aggregate PGA memory available to all server processes attached to the instance. See Also: "Memory Configuration and Use" in the Oracle Database Performance Tuning Guide for information on PGA memory management.
`PROCESSES` | Sets the maximum number of operating system processes that can be connected to Oracle concurrently. The value of this parameter must be 6 or greater (5 for the background processes plus 1 for each user process). For example, if you plan to have 50 concurrent users, set this parameter to at least 55. Many other initialization parameter values are deduced from this value.
`SGA_MAX_SIZE` | This parameter is the maximum size of the SGA for a running instance. Set this parameter to the amount of memory that you want dedicated for the SGA, which includes the following memory pools: Database buffer cache Shared pool Large pool Java pool It is a good practice to regularly monitor the buffer cache hit ratio and size the SGA so that the buffer cache has an adequate number of frames for the workload. The buffer cache hit ratio may be calculated from data in the view `V$SYSSTAT`. Also the view `V$DB_CACHE_ADVICE` provides data that can be used to tune the buffer cache. See Also: the chapter, "Memory Configuration and Use" in the Oracle Database Performance Tuning Guide for detailed information on how to set the `SGA_MAX_SIZE` parameter, including on how to use the `V$SYSSTAT` and `V$DB_CACHE_ADVICE` views to optimize the buffer cache hit ratio.
`SGA_TARGET` | Setting this parameter to a nonzero value enables Automatic Shared Memory Management. Oracle strongly recommends the use of automatic memory management, both to simplify configuration and to improve performance. Automatic Shared Memory Management was introduced with the Oracle Database 10g (10.1). For prior versions, you must manually configure individual SGA memory pools. See Also: the section, "Automatic Shared Memory Management" in the Chapter, "Memory Configuration and Use" in the Oracle Database Performance Tuning Guide for details on chosing a value for the `SGA_TARGET` parameter.
`UNDO_TABLESPACE` `UNDO_MANAGEMENT` | Oracle strongly recommends that you use automatic undo management (`UNDO_MANAGEMENT = AUTO`) and manage undo space using an `UNDO_TABLESPACE`. For backward compatibility reasons, the default value of `UNDO_MANAGEMENT` is `MANUAL`. See Also: Oracle Database Performance Tuning Guide for additional information on undo space management.

#### 3.1.7.2 Tuning Redo Logs Location and Sizing

Managing the database I/O load balancing is a non-trivial task. However, tuning the redo log options can provide performance improvement for applications running in an Oracle Application Server environment, and in some cases, you can significantly improve I/O throughput by moving the redo logs to a separate disk.

The size of the redo log files can also influence performance, because the behavior of the database writer and archiver processes depend on the redo log sizes. Generally, larger redo log files provide better performance by reducing checkpoint activity. It is not possible to provide a specific size recommendation for redo log files, but redo log files in the range of a hundred megabytes to a few gigabytes are considered reasonable. Size your online redo log files according to the amount of redo your system generates. A rough guide is to switch logs at most once every twenty minutes. Set the initialization parameter `LOG_CHECKPOINTS_TO_ALERT = true` to have checkpoint times written to the alert file.

The complete set of required redo log files can be created during database creation. After they are created, the size of a redo log size cannot be changed. However, new, larger files can be added later, and the original (smaller) ones can subsequently be dropped.

See Also: The chapters, "Configuring a Database for Performance" and "I/O Configuration and Design" in the Oracle Database Performance Tuning Guide

#### 3.1.7.3 Automatic Segment-Space Management (ASSM)

For permanent tablespaces, Oracle recommends using automatic segment-space management. Such tablespaces, often referred to as bitmap tablespaces, are locally managed tablespaces with bitmap segment space management.

For backward compatibility, the default local tablespace segment-space management mode is `MANUAL`.

See Also: Oracle Database Concepts for a discussion of free space management, and Oracle Database Administrator's Guide for more information on creating and using automatic segment-space management for tablespaces.

### 3.1.8 Verify Logging Levels

You need to assure that application and server logging levels are set appropriately, and that debugging properties or other application level debugging flags are set to appropriate levels or disabled. Set Oracle Application Server OC4J logger log levels to log messages at the `INFO` level (do not set log levels to levels that produce more diagnostic message, including the `FINE` or `TRACE` levels).

To configure OC4J component loggers through Application Server Control Console, do the following from the OC4J home page:

1. Click the Administration link

2. Click Logger Configuration under Properties.

3. Set the root Log Level to the desired value, or expand the tree to select individual Log Levels for specified loggers.

4. Click Apply to apply your changes to the OC4J runtime.

### 3.1.9 Reuse EJB Instances

This section describes EJB options for creating and reusing instances that you can tune to improve EJB performance; these attributes are specific to OC4J and apply for all types of EJBs (except Stateful Session EJBs). You can configure these options by setting attributes in `orion-ejb-jar.xml`.

The `min-instances` attribute specifies the minimum number of bean implementation instances to be kept instantiated or pooled. The default value is 0. For best performance, you should specify a value for `min-instances` other than 0. If `min-instances is > 0`, OC4J maintains the `min-instances` number of instances in the pool when they are not in use. For instances above the `min-instances`, the instances are removed from the pool after the `pool-cache-timeout` specified timeout expires.

The default value for the `pool-cache-timeout` is 60 seconds, which is typically too low for frequently accessed EJBs. If the pool-cache-timeout is 0 or negative, then the `pool-cache-timeout` is disabled and beans are not removed from the pool.

For performance tuning, try to reduce the frequency of the removal of beans from the pool by setting the `pool-cache-timeout` to a large value. You should set the `pool-cache-timeout` to a large enough value to allow OC4J to avoid destroying and then re-creating instances while your J2EE application is running.

## 3.2 Advanced Performance Areas

This section describes areas that can provide improved performance for some usage cases and environments.

This section covers the following topics:

- Managing Concurrency and Limiting Connections

- Load Balancing

- Using the -XX:AppendRatio Option (Sun JVM argument)

### 3.2.1 Managing Concurrency and Limiting Connections

Oracle Application Server lets you limit concurrency at multiple layers of the system to match specific usage needs. In addition to controlling HTTP connections, you can control concurrency at additional levels of the product to meet specific usage requirements.

This section covers the following topics:

- Using OC4J Thread Pools to Control Concurrency

- Setting the Maximum Number of Connections for Data Sources

- Controlling the Number of EJB Instances When Using EJBs

- Limiting Remote EJB Client Connections

See Also: "Specify Sufficient Oracle HTTP Server Connections"

#### 3.2.1.1 Using OC4J Thread Pools to Control Concurrency

By default, OC4J creates two thread pools, as described in the following list. New threads are created and added to the pools on an as-needed basis. You can limit the number of threads that OC4J creates for the tasks required to handle application requests. The default behavior should be sufficient for most common usage scenarios.

- 2 Thread Pool Configuration

  Application Server Thread Pool contains the pool of worker threads plus the threads used for RMI server connection threads. The default behavior allows unbounded threads and threads are created on demand.

  Work Management Thread Pool contains the pool of threads that OC4J reserves specifically for use by deployed resource adapters.

Alternatively, you can divide the Application Server Thread Pool into two pools to segregate the RMI connection threads into their own pool. In this case, the Application Server Thread Pool is divided into a worker thread pool and a connection thread pool.

- 3 Thread Pool Configuration

  Application Server Thread Pools

  - Worker Thread Pool contains the pool of worker threads

  - Connection Thread Pool contains the RMI server connection threads

  Work Management Thread Pool contains the pool of threads that OC4J reserves specifically for use by deployed resource adapters.

Note: Using the thread pool management options is considered an expert-mode task. When specifying thread pool settings, it is important to remember that the resulting concurrency will be determined by the sum of all the active thread pools in the OC4J process.

Specifying thread pool options can improve performance in certain cases, including the following:

- A hardware resource is nearing maximum capacity on Oracle Application Server 10g. For example, consider the case where a site has a user population of 1000 users with the site often processing 20 concurrent requests; at this rate, the site is nearing full resource utilization. Occasionally, the site needs to handle peaks of 75 to 100 concurrent requests. Specifying a thread pool with a maximum number of threads value in the range of 20 to 25 is likely to yield better overall results (setting the value in the range of 75 to 100 is unlikely to improve peak performance, since at this level there are no additional resources available to service the threads).

- A hardware resource is nearing maximum capacity on the database or the database is a bottleneck due to other limitations, and you want to limit connections to the database using OC4J thread control (in addition, you can limit database connections using the data-source `max-connections` parameter).

Thread pool tuning does not improve performance in the following situations:

- There are sufficient hardware resources available on the system when the system is under maximum load and there are no other known software issues such as database locking problems. This includes both Oracle Application Server 10g tier and database systems. For example, consider the case where a site has a user population of 1000 users but typically only sees a few concurrent requests. In this case, using application server thread pool tuning to limit threads is not advantageous, since the number of threads is determined by the concurrent request rate, which for this case is very low.

- There already are sufficient concurrency limits specified elsewhere in Oracle Application Server 10g or in the database systems. For example when the Oracle HTTP Server `MaxClients` directive is set to control concurrency, at the HTTP server level, or when the data source `max-connections` attribute is set to control concurrency for connections to the database.

Note: If the number of incoming requests is consistently higher than the request rate that the physical hardware can support, consider increasing the site's physical resources. Likewise, if the response time at peak periods is unacceptable, then you may need to increase the hardware configuration to remedy this situation.

See Also: "Specify Sufficient Oracle HTTP Server Connections" "Setting the Maximum Number of Connections for Data Sources"

##### 3.2.1.1.1 Controlling and Using Application Server Thread Pools

This section describes how to manage thread settings for a 2-Pool configuration. See "Using the Work Manager Thread Pool" for information on options for specifying limitations for the Work Manager Thread Pool.

You can manage application server thread pools through any of the following:

- Adding the `<global-thread-pool>` element with appropriate attributes set in `server.xml`. For example,

      <global-thread-pool min="30" max="30" queue="60" \>

- Updating the attributes in the `ApplicationServerThreadPool` MBean, which is accessible through the system MBean Browser in Application Server Control Console.

- Using the Thread Pool Configuration page in Application Server Control Console. You access this page by starting at the Administration secondary tab and then clicking the Thread Pool Configuration task (see Figure 3-1).

Figure 3-1 Application Server Control Console Thread Pool Configuration

Description of "Figure 3-1 Application Server Control Console Thread Pool Configuration"

Notes for specifying thread pool options:

1. Oracle recommends setting the `min` value (Minimum Pool Size) equal to the `max` value (Maximum Pool Size) for the simple case. When `min`=`max`, you should set the `queue` to a value equal to the maximum concurrent number of requests you expect. For example, if you are using Oracle HTTP Server with one OC4J instance and no direct RMI connections, then the value of `MaxClients` would represent your maximum concurrency. Note: if you set `MaxClients` to a very large number, a setting of 300 or less for the `queue` size is probably sufficient. See "Validating and Monitoring Thread Pool Performance" for instructions on monitoring your thread pool and queues.

   Oracle recommends starting with a smaller number of threads, for example set `min` and `max` to a value in the range of 15 to 40. Monitor your resulting performance with this value. If the site has sufficient CPU and memory resources available, and the concurrent request rate is higher than the current thread count setting, try increasing the number of threads. If you observe that the system is resource saturated, then try reducing the number of threads. Specific thread settings depend on the characteristics of your application.

2. The simplest case is to set `min` value = `max` value. However, if you do set a larger `max` value to accommodate occasional peaks of traffic, OC4J will add a new thread on each request until the minimum number of threads are created. After the minimum number of threads are created, new threads will not be created until the queue is full. OC4J attempts to keep the number of threads at or near the minimum, unless the queue is full. If you set the `min` value less than the `max` value, it is generally advisable to keep the queue size small; the `queue` default value is 80. If you do not see your threads increasing beyond the minimum value, then you should decrease the queue size.

See Also: Oracle Containers for J2EE Configuration and Administration Guide for more details on using the application server thread pool

##### 3.2.1.1.2 Using the Connection Thread Pool

The application server thread pool supports a second thread pool for RMI connection threads. By default the RMI connection threads are allocated from the application server worker thread pool. To create two pools you configure the `min`, `max`, `queue`, and `keepAlive` attributes and the `cx-min`, `cx-max`, `cx-queue`, and `cx-keepAlive` attributes in the `<global-thread-pool>` element.

Defining the second pool using the `cx-` attributes puts a separate limit on the RMI connections; they are not taken from the worker thread pool. In either case, the work from the RMI connections executes from threads in the application server worker thread pool. Specify the connection pool to separate potentially long-lived connection threads from the threads used for application work. This frees the threads in the worker thread pool to do work, instead of being allocated to long lived connections.

Note: Specifying the second thread pool, the connection pool, is only needed if you anticipate many concurrent RMI connections, where the number varies greatly and you are trying to bound the active worker threads.

See Also: Section, "Managing Thread Pool Configurations", in Chapter 11, Task Manager and Thread Pool Configuration, in the Oracle Containers for J2EE Configuration and Administration Guide

##### 3.2.1.1.3 Using the Work Manager Thread Pool

Starting with 10g Release 3 (10.1.3), EJBs of type MDB use receiver threads with the JMS Resource Adapter. OC4J allocates these threads from the `work-manager-thread-pool` (not from the application server thread pool). However, you need to consider receiver threads to control the overall concurrency on the system. Oracle recommends leaving the `work-manager-thread-pool` at the default setting (by default `work-manager-thread-pool` threads are unlimited). If you want to control concurrency for EJB MDBs, use the JMS ReceiverThreads maximum values. The overall concurrency limit on your system includes the `global-thread-pool` max threads plus the sum of all the MDB receiverThreads configured for your applications deployed to OC4J, or the `max` `work-manager-thread-pool` setting if smaller.

##### 3.2.1.1.4 Validating and Monitoring Thread Pool Performance

You can use the Application Server Control Console Current Pool Size and Current Queue Size metrics to observe OC4J thread use on your system. You can access these metrics by starting at the Administration secondary tab and then clicking the Thread Pool Configuration task (see Figure 3-1). The Current Pool Size shows the current number of threads in the pool. The Current Queue Size shows the current number of requests waiting in the queue for a thread to become available. Oracle recommends starting with the default settings and observing the behavior on your system. You should also observe these metrics after setting any thread pool or queue size limits for your instance.

#### 3.2.1.2 Setting the Maximum Number of Connections for Data Sources

For applications that use a database, performance can improve when the connection pool associated with a data source limits the number of connections. You can use the `max-connection`s attribute to limit the database requests from Oracle Application Server so that incoming requests do not saturate the database, or to limit the database requests so that the database access does not overload Oracle Application Server-tier resource.

The connection pool `max-connections` attribute specifies the maximum number of connections that a connection pool allows. By default, the value of `max-connections` is set to -1 (unlimited). For best performance, you should specify a value for `max-connections` that matches the number appropriate to your database performance characteristics.

Limiting the total number of open database connections to a number your database can handle is an important tuning consideration. You should check to make sure that your database is configured to allow at least as large a number of open connections as the total of the values specified for all the data sources `max-connections` option, as specified in all the applications that access the database.

#### 3.2.1.3 Controlling the Number of EJB Instances When Using EJBs

You may want to limit the number of EJB instances to reduce memory usage or to control concurrency to reduce contention on resources that the EJBs use (for example a data source).

The `max-instances` parameter specifies the number of bean instances allowed in memory – either instantiated or pooled.

- For all types of EJBs except stateful session beans, when the `max-instances` value is reached, and a new EJB is requested, the container waits the number of milliseconds set in the `call-timeout` attribute to see if a bean instance becomes available in the pool. If no bean instance is available in the pool then a `TimeoutExpiredException` is thrown back to the client.

- For stateful session beans, when the `max-instances` value is reached, the container attempts to passivate the oldest bean instance from memory. If unsuccessful, the container waits the number of milliseconds set in the `call-timeout` attribute to see if a bean instance is removed from memory, either through passivation, using the `remove()` method, or by bean expiration before a `TimeoutExpiredException` is thrown back to the client.

To allow an unlimited number of bean instances, use `max-instances` = 0 (the default value is 0).

Set `max-instances` < 0, for example to -1, to disable instance pooling. In this case OC4J creates a new bean instance or context when starting the EJB call, and releases the context and throws the instance away to `Non-existence` state at the end of the call.

The exception, `com.evermind.server.ejb.TimeoutExpiredException`: `timeout expired waiting for an instance`, occurs when there is no available EJB instance. To avoid this problem set the `max-instances` and `call-timeout` parameters appropriately.

#### 3.2.1.4 Limiting Remote EJB Client Connections

To limit remote EJB client connections, you can use the global thread pool features that control the maximum number of threads that service incoming EJB clients. By configuring the `<global-thread-pool>` in `server.xml` to use two thread pools, you can set the parameter `cx-max` to limit remote EJB client connections.

### 3.2.2 Load Balancing

Oracle Application Server provides load-balancing features that spread the J2EE application load and incoming requests among multiple application server instances, which generally results in higher throughput and shorter response time. Using multiple application server instances with load-balancing allows you to improve performance by directing requests across the multiple application server instances. In addition, you can use multiple application server instances running on multiple hosts to handle high availability and failover needs.

This section covers the following topics:

- Configuring Multiple Oracle Application Server Instances

- Web Application Load Balancing

- EJB Application Load Balancing

Note: The Oracle Application Server features that provide replication for failover with Web sessions and for stateful session EJBs have a performance overhead; only use these features if you need replication for failover.

#### 3.2.2.1 Configuring Multiple Oracle Application Server Instances

This section covers the following:

- Determining the Number of OC4J Processes

- Partitioning Applications into Different OC4J Instances

##### 3.2.2.1.1 Determining the Number of OC4J Processes

Determining the optimal ratio of OC4J processes to available CPUs is dependent on the characteristics of the applications you run, the OC4J configuration, the hardware configuration, and the type and number of expected incoming requests. In hardware configurations with a small number of CPUs, you may only need one OC4J instance.

Adding OC4J instances beyond the available resources of the system does not improve performance. For example, if one OC4J instance is sufficient to saturate the CPU resources of a system, adding additional OC4J instances is not likely to improve performance and may, in fact, degrade it. A good starting point is to initially configure one OC4J instance and measure the performance improvement from adding additional OC4J instances.

See Also: Oracle Application Server High Availability Guide Oracle Containers for J2EE Configuration and Administration Guide

##### 3.2.2.1.2 Partitioning Applications into Different OC4J Instances

Partitioning different applications to be run under different OC4J instances, each of which has different requirements, may help improve the performance of your applications. In this case, you may want to configure different OC4J instances to service the different applications. After deploying the applications to different OC4J instances, you can monitor the performance to see if the overall throughput increases, or the response time decreases.

#### 3.2.2.2 Web Application Load Balancing

In an Oracle Application Server environment, the Oracle HTTP Server uses `mod_oc4j` to load balance requests between the available OC4J instances. In this environment you can select `mod_oc4j` configuration options to choose the appropriate load balancing policies to improve performance. By default, the requests are routed using a roundrobin algorithm.

At many sites Oracle Application Server uses the Oracle HTTP Server module `mod_oc4j` to load balance incoming stateless HTTP requests. By selecting the appropriate load balancing policy for `mod_oc4j` you can improve performance on your site.The `mod_oc4`j module supports several configurable load balancing policies, including the following:

- Round robin routing (this is the default `mod_oc4j` load balancing policy)

- Random routing

- Round robin or random with local affinity routing, using the `local` option

- Round robin or random with host-level weighted routing, using the `weighted` option

Note: For a session based request `mod_oc4j` always directs the request to the original OC4J which created the session, unless the original OC4J process is not available. In case of failure, `mod_oc4j` sends the request to another OC4J within the same group as the original request (either within same host if available, or on a different host).

Recommendations for Load Balancing with `mod_oc4j`:

1. On a Single Host for both Oracle HTTP Server and OC4J, the default load balancing policy, round robin load balancing is recommended. Random load balancing typically gives comparable performance.

2. With Oracle HTTP Server on a separate host from OC4J:

   - Using a Single OC4J host the default load balancing policy, round robin load balancing is recommended. Random load balancing typically gives comparable performance.

   - Using Multiple OC4J hosts, if all OC4J hosts provide comparable capacity, the default load balancing is recommended. Note that the number of OC4J processes started on a host will implicitly weight the number of requests sent to that host even with random or round robin load balancing. A host with 4 OC4J processes will receive 4 times as many requests as a host with 1 OC4J process.

   - With Multiple OC4J Hosts with varying hardware resources or capacity, you may wish to weight the number of requests sent to each host explicitly to match its capacity. In this case, use either the round robin with weighted option or the random with weighted option. If the hosts have comparable capacity, use simple random or round robin load balancing.

     For example, to configure the `mod_oc4j` module in Oracle HTTP Server to specify round robin with a routing weight of 3 for `Host_A` and a routing weight of 1 for `Host_B`, add the following directives to `mod_oc4j.conf`:

         Oc4jSelectMethod roundrobin:weighted
         Oc4jRoutingWeight   Host_A 3

     In this example, you do not need to specify a routing weight for `Host_B`, since the default routing weight is 1.

3. With Multiple hosts with Oracle HTTP Server and OC4J on each host:

   - Multiple hosts with Oracle HTTP Server and OC4J on each host, and a hardware load balancer. Select the local affinity option to direct `mod_oc4j` to only select the local OC4J processes to service incoming requests. This will generally improve performance. When no local OC4J processes are available, `mod_oc4j` selects from the list of available remote OC4J processes.

     For example, to select the round robin policy with local affinity, specify the following directive in `mod_oc4j.conf`:

         Oc4jSelectMethod roundrobin:local

#### 3.2.2.3 EJB Application Load Balancing

After an EJB application is deployed to multiple OC4J instances, an EJB client-side application can load balance its requests across the available OC4J instances. To use load balancing, the client-side application configures the JNDI properties to use load balancing. For good performance in some clients, you need to set `oracle.j2ee.rmi.loadBalance=context` to load balance for every initialcontext call, rather than only once for the entire client.

See Also: "Configuring ORMI Request Load Balancing" in the Oracle Containers for J2EE Services Guide "Understanding OC4J EJB Application Clustering Services" in the Oracle Containers for J2EE Enterprise JavaBeans Developer's Guide

### 3.2.3 Using the -XX:AppendRatio Option (Sun JVM argument)

With the Sun 5.0 JVM on Linux, under some circumstances under heavy load, synchronization in an application can result in thread starvation. This may cause some requests for your application to appear hung or to timeout after a long time.

If you believe your installation has this problem, you can also try the following JDK parameter: `-XX:AppendRatio=3`. This setting will help to balance the load, but may decrease your overall system throughput. We only recommend this setting for installations experiencing problems due to this bug.

See Also: See the SUN bug database for a description of this issue and the suggested workaround: `http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4985566`