6 Tuning WebLogic Server

On-demand Deployment of Internal Applications

WebLogic Server deploys many internal applications during startup. Many of these internal applications are not needed by every user. You can configure WebLogic Server to wait and deploy these applications on the first access (on-demand) instead of always deploying them during server startup. This can conserve memory and CPU time during deployment as well as improving startup time and decreasing the base memory footprint for the server.

WebLogic Server deploys many internal applications during startup. Many of these internal applications are not needed by every user. You can configure WebLogic Server to wait and deploy these applications on the first access (on-demand) instead of always deploying them during server startup. This can conserve memory and CPU time during deployment as well as improving startup time and decreasing the base memory footprint for the server.For a development domain, the default is for WLS to deploy internal applications on-demand. For a production-mode domain, the default is for WLS to deploy internal applications as part of server startup. For more information on how to use and configure this feature, see On-demand Deployment of Internal Applications in Deploying Applications to WebLogic Server.

Use FastSwap Deployment to Minimize Redeployment Time

In deployment mode, you can set WebLogic Server to redefine Java classes in-place without reloading the ClassLoader. This means that you do not have to wait for an application to redeploy and then navigate back to wherever you were in the Web page flow. Instead, you can make your changes, auto compile, and then see the effects immediately. For more information on how to use and configure this feature, see Using FastSwap Deployment to Minimize Redeployment in Deploying Applications to WebLogic Server.

Generic Overrides

Generic overrides allow you to override application specific property files without having to crack a jar file by placing application specific files to be overridden into the AppFileOverrides optional subdirectory. For more information on how to use and configure this feature, see Generic File Loading Overrides in Deploying Applications to WebLogic Server.

Tuning a Work Manager

In this release, WebLogic Server allows you to configure how your application prioritizes the execution of its work. Based on rules you define and by monitoring actual runtime performance, WebLogic Server can optimize the performance of your application and maintain service level agreements (SLA).

You tune the thread utilization of a server instance by defining rules and constraints for your application by defining a Work Manger and applying it either globally to WebLogic Server domain or to a specific application component. The primary tuning considerations are:

• How Many Work Managers are Needed?

• What are the SLA Requirements for Each Work Manager?

See "Using Work Managers to Optimize Scheduled Work" in Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server.

How Many Work Managers are Needed?

Each distinct SLA requirement needs a unique work manager.

What are the SLA Requirements for Each Work Manager?

Service level agreement (SLA) requirements are defined by instances of request classes. A request class expresses a scheduling guideline that a server instance uses to allocate threads. See "Understanding Work Managers" in Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server.

Tuning Execute Queues

In previous versions of WebLogic Server, processing was performed in multiple execute queues. Different classes of work were executed in different queues, based on priority and ordering requirements, and to avoid deadlocks. See Appendix B, "Using the WebLogic 8.1 Thread Pool Model.".

Understanding the Differences Between Work Managers and Execute Queues

The easiest way to conceptually visualize the difference between the execute queues of previous releases with work managers is to correlate execute queues (or rather, execute-queue managers) with work managers and decouple the one-to-one relationship between execute queues and thread-pools.

For releases prior to WebLogic Server 9.0, incoming requests are put into a default execute queue or a user-defined execute queue. Each execute queue has an associated execute queue manager that controls an exclusive, dedicated thread-pool with a fixed number of threads in it. Requests are added to the queue on a first-come-first-served basis. The execute-queue manager then picks the first request from the queue and an available thread from the associated thread-pool and dispatches the request to be executed by that thread.

For releases of WebLogic Server 9.0 and higher, there is a single priority-based execute queue in the server. Incoming requests are assigned an internal priority based on the configuration of work managers you create to manage the work performed by your applications. The server increases or decreases threads available for the execute queue depending on the demand from the various work-managers. The position of a request in the execute queue is determined by its internal priority:

• The higher the priority, closer it is placed to the head of the execute queue.

• The closer to the head of the queue, more quickly the request will be dispatched a thread to use.

Work managers provide you the ability to better control thread utilization (server performance) than execute-queues, primarily due to the many ways that you can specify scheduling guidelines for the priority-based thread pool. These scheduling guidelines can be set either as numeric values or as the capacity of a server-managed resource, like a JDBC connection pool.

Migrating from Previous Releases

If you upgrade application domains from prior releases that contain execute queues, the resulting 9.x domain will contain execute queues.

• Migrating application domains from a previous release to WebLogic Server 9.x does not automatically convert an execute queues to work manager.

• If execute queues are present in the upgraded application configuration, the server instance assigns work requests appropriately to the execute queue specified in the dispatch-policy.

• Requests without a dispatch-policy use the self-tuning thread pool.

See "Roadmap for Upgrading Your Application Environment" in Oracle Fusion Middleware Upgrade Guide for Oracle WebLogic Server.

Tuning the Stuck Thread Detection Behavior

WebLogic Server automatically detects when a thread in an execute queue becomes "stuck." Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread.

WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads. Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the "warning" and "critical" health states when all threads in a particular execute queue become stuck. For more information, see "Configuring WebLogic Server to Avoid Overload Conditions" in Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server. To configure stuck thread detection behavior, see "Tuning execute thread detection behavior" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Tuning Muxers

WebLogic Server uses software modules called muxers to read incoming requests on the server and incoming responses on the client. These muxers are of two primary types: the Java muxer or native muxer.

A Java muxer has the following characteristics:

• Uses pure Java to read data from sockets.

• It is also the only muxer available for RMI clients.

• Blocks on reads until there is data to be read from a socket. This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets. This is typically not an issue for clients, but it can create a huge bottleneck for a server.

Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use. See "Enable native IO" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer. This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high. Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers, the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console. See Changing the Number of Available Socket Readers. Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size. Each thread waits for a fixed amount of time for data to become available at a socket. If no data arrives, the thread moves to the next socket.

With native muxers, you may be able to improve throughput for some cpu-bound applications (for example, SpecJAppServer) by using the following:

-Dweblogic.socket.SocketMuxer.DELAY_POLL_WAKEUP=xx

where xx is the amount of time, in microseconds, to delay before checking if data is available. The default value is 0, which corresponds to no delay.

Which Platforms Have Performance Packs?

Benchmarks show major performance improvements when you use native performance packs on machines that host WebLogic Server instances. Performance packs use a platform-optimized, native socket multiplexor to improve server performance. For example, the native socket reader multiplexor threads have their own execute queue and do not borrow threads from the default execute queue, which frees up default execute threads to do application work

To see which platforms currently have performance packs available:

1. Go to the Certifications Pages at http://www.oracle.com/technology/software/products/ias/files/fusion_certification.html.

2. Select your platform from the list of certified platforms.

3. Use your browser's Edit > Find to locate all instances of "Performance Pack" to verify whether it is included for the platform.

Enabling Performance Packs

The use of NATIVE performance packs are enabled by default in the configuration shipped with your distribution. You can use the Administration Console to verify that performance packs are enabled. See "Enable native IO" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Changing the Number of Available Socket Readers

If you must use the pure-Java socket reader implementation for host machines, you can improve the performance of socket communication by configuring the proper number of socket reader threads for each server instance and client machine. See "Tuning the number of available socket readers" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Network Channels

Network channels, also called network access points, allow you to specify different quality of service (QOS) parameters for network communication. Each network channel is associated with its own exclusive socket using a unique IP address and port. By default, requests from a multi-threaded client are multiplexed over the same remote connection and the server instance reads requests from the socket one at a time. If the request size is large, this becomes a bottleneck.

Although the primary role of a network channel is to control the network traffic for a server instance, you can leverage the ability to create multiple custom channels to allow a multi-threaded client to communicate with server instance over multiple connections, reducing the potential for a bottleneck. To configure custom multi-channel communication, use the following steps:

1. Configure multiple network channels using different IP and port settings. See "Configure custom network channels" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

2. In your client-side code, use a JNDI URL pattern similar to the pattern used in clustered environments. The following is an example for a client using two network channels:

   t3://<ip1>:<port1>,<ip2>:<port2>
   

See "Understanding Network Channels" in Oracle Fusion Middleware Configuring Server Environments for Oracle WebLogic Server.

Reducing the Potential for Denial of Service Attacks

To reduce the potential for Denial of Service (DoS) attacks while simultaneously optimizing system availability, WebLogic Server allows you to specify the following settings:

• Maximum incoming message size

• Complete message timeout

• Number of file descriptors (UNIX systems)

For optimal system performance, each of these settings should be appropriate for the particular system that hosts WebLogic Server and should be in balance with each other, as explained in the sections that follow.

Tune the Chunk Parameters

A chunk is a unit of memory that the WebLogic Server network layer, both on the client and server side, uses to read data from and write data to sockets. To reduce memory allocation costs, a server instance maintains a pool of these chunks. For applications that handle large amounts of data per request, increasing the value on both the client and server sides can boost performance. The default chunk size is about 4K. Use the following properties to tune the chunk size and the chunk pool size:

• weblogic.Chunksize—Sets the size of a chunk (in bytes). The primary situation in which this may need to be increased is if request sizes are large. It should be set to values that are multiples of the network's maximum transfer unit (MTU), after subtracting from the value any Ethernet or TCP header sizes. Set this parameter to the same value on the client and server.

• weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. The default value is 2048. The value may need to be increased if the server starts to allocate and discard chunks in steady state. To determine if the value needs to be increased, monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk.

• weblogic.PartitionSize—Sets the number of pool partitions used (default is 4). The chunk pool can be a source of significant lock contention as each request to access to the pool must be synchronized. Partitioning the thread pool spreads the potential for contention over more than one partition.

Tuning Connection Backlog Buffering

You can tune the number of connection requests that a WebLogic Server instance will accept before refusing additional requests. The Accept Backlog parameter specifies how many Transmission Control Protocol (TCP) connections can be buffered in a wait queue. This fixed-size queue is populated with requests for connections that the TCP stack has received, but the application has not accepted yet. For more information on TCP tuning, see Basic OS Tuning Concepts.

You can tune the number of connection requests that a WebLogic Server instance will accept before refusing additional requests, see "Tune connection backlog buffering" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Compiling EJB Classes

Use the weblogic.appc utility to compile EJB container classes. If you compile Jar files for deployment into the EJB container, you must use weblogic.appc to generate the container classes. By default, ejbc uses the javac compiler. You may be able to improve performance by specifying a different compiler (such as IBM Jikes) using the -compiler flag or using the Administration console. For more information, see:

• "Implementing Enterprise Java Beans" in Oracle Fusion Middleware Programming Enterprise JavaBeans for Oracle WebLogic Server.

• "Configure compiler options" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Setting JSP Compiler Options

The following sections provide information on tuning JSP compiler options:

• Precompile JSPs

• Optimize Java Expressions

failed: java.io.IOException: Not enough space

Scalability and High Availability

Scalability is the ability of a system to grow in one or more dimensions as more resources are added to the system. Typically, these dimensions include (among other things), the number of concurrent users that can be supported and the number of transactions that can be processed in a given unit of time.

Given a well-designed application, it is entirely possible to increase performance by simply adding more resources. To increase the load handling capabilities of WebLogic Server, add another WebLogic Server instance to your cluster—without changing your application. Clusters provide two key benefits that are not provided by a single server: scalability and availability.

WebLogic Server clusters bring scalability and high-availability to Java EE applications in a way that is transparent to application developers. Scalability expands the capacity of the middle tier beyond that of a single WebLogic Server or a single computer. The only limitation on cluster membership is that all WebLogic Servers must be able to communicate by IP multicast. New WebLogic Servers can be added to a cluster dynamically to increase capacity.

A WebLogic Server cluster guarantees high-availability by using the redundancy of multiple servers to insulate clients from failures. The same service can be provided on multiple servers in a cluster. If one server fails, another can take over. The ability to have a functioning server take over from a failed server increases the availability of the application to clients.

Clustering in the Messaging Service is provided through distributed destinations; connection concentrators, and connection load-balancing (determined by connection factory targeting); and clustered Store-and-Forward (SAF). Client load-balancing with respect to distributed destinations is tunable on connection factories. Distributed destination Message Driven Beans (MDBs) that are targeted to the same cluster that hosts the distributed destination automatically deploy only on cluster servers that host the distributed destination members and only process messages from their local destination. Distributed queue MDBs that are targeted to a different server or cluster than the host of the distributed destination automatically create consumers for every distributed destination member. For example, each running MDB has a consumer for each distributed destination queue member.

How to Ensure Scalability for WebLogic Clusters

In general, any operation that requires communication between the servers in a cluster is a potential scalability hindrance. The following sections provide information on issues that impact the ability to linearly scale clustered WebLogic servers:

• Database Bottlenecks

• Session Replication

• Asynchronous HTTP Session Replication

• Invalidation of Entity EJBs

• Invalidation of HTTP sessions

• JNDI Binding, Unbinding and Rebinding

Database Bottlenecks

In many cases where a cluster of WebLogic servers fails to scale, the database is the bottleneck. In such situations, the only solutions are to tune the database or reduce load on the database by exploring other options. See Chapter 8, "DataBase Tuning" and Chapter 11, "Tuning JDBC Applications".

Session Replication

User session data can be stored in two standard ways in a Java EE application: stateful session EJBs or HTTP sessions. By themselves, they are rarely a impact cluster scalability. However, when coupled with a session replication mechanism required to provide high-availability, bottlenecks are introduced. If a Java EE application has Web and EJB components, you should store user session data in HTTP sessions:

• HTTP session management provides more options for handling fail-over, such as replication, a shared DB or file.

• Superior scalability.

• Replication of the HTTP session state occurs outside of any transactions. Stateful session bean replication occurs in a transaction which is more resource intensive.

• The HTTP session replication mechanism is more sophisticated and provides optimizations a wider variety of situations than stateful session bean replication.

See Session Management.

Asynchronous HTTP Session Replication

Asynchronous replication of http sessions provides the option of choosing asynchronous session replication using:

• Asynchronous HTTP Session Replication using a Secondary Server

• Asynchronous HTTP Session Replication using a Database

Invalidation of Entity EJBs

This applies to entity EJBs that use a concurrency strategy of Optimistic or ReadOnly with a read-write pattern.

Optimistic—When an Optimistic concurrency bean is updated, the EJB container sends a multicast message to other cluster members to invalidate their local copies of the bean. This is done to avoid optimistic concurrency exceptions being thrown by the other servers and hence the need to retry transactions. If updates to the EJBs are frequent, the work done by the servers to invalidate each other's local caches become a serious bottleneck. A flag called cluster-invalidation-disabled (default false) is used to turn off such invalidations. This is set in the rdbms descriptor file.

ReadOnly with a read-write pattern—In this pattern, persistent data that would otherwise be represented by a single EJB are actually represented by two EJBs: one read-only and the other updatable. When the state of the updateable bean changes, the container automatically invalidates corresponding read-only EJB instance. If updates to the EJBs are frequent, the work done by the servers to invalidate the read-only EJBs becomes a serious bottleneck.

Invalidation of HTTP sessions

Similar to Invalidation of Entity EJBs, HTTP sessions can also be invalidated. This is not as expensive as entity EJB invalidation, since only the session data stored in the secondary server needs to be invalidated. Oracle advises users to not invalidate sessions unless absolutely required.

JNDI Binding, Unbinding and Rebinding

In general, JNDI binds, unbinds and rebinds are expensive operations. However, these operations become a bigger bottleneck in clustered environments because JNDI tree changes have to be propagated to all members of a cluster. If such operations are performed too frequently, they can reduce cluster scalability significantly.

Running Multiple Server Instances on Multi-CPU Machines

With multi-processor machines, additional consideration must be given to the ratio of the number of available CPUs to clustered WebLogic Server instances. Because WebLogic Server has no built-in limit to the number of server instances that reside in a cluster, large, multi-processor servers, such as Sun Microsystems' Sun Enterprise 10000, can potentially host very large clusters or multiple clusters.

In order to determine the optimal ratio of CPUs to WebLogic server instances, you must first ensure that an application is truly CPU-bound, rather than network or disk I/O-bound. Use the following steps to determine the optional ratio of CPUs to server instances:

1. Test your application to determine the Network Requirements.

   If you discover that an application is primarily network I/O-bound, consider measures to increase network throughput before increasing the number of available CPUs. For truly network I/O-bound applications, installing a faster network interface card (NIC) may increase performance more than additional CPUs, because most CPUs would remain idle while waiting to read available sockets.

2. Test your application to determine the Disk I/O Requirements.

   If you discover that an application is primarily disk I/O-bound, consider upgrading the number of disk spindles or individual disks and controllers before allocating additional CPUs.

3. Begin performance tests using a ratio of one WebLogic Server instance for every available CPU.

4. If CPU utilization is consistently at or near 100 percent, increase the ratio of CPUs to server instances by adding an additional CPU. Add additional CPUs until utilization reaches an acceptable level. Remember, always reserve some spare CPU cycles on your production systems to perform any administration tasks that may occur.

Using the Administration Console to Monitor WebLogic Server

The tool for monitoring the health and performance of your WebLogic Server domain is the Administration Console. See "Monitor servers" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help.

Using the WebLogic Diagnostic Framework

The WebLogic Diagnostic Framework (WLDF) is a monitoring and diagnostic framework that defines and implements a set of services that run within WebLogic Server processes and participate in the standard server life cycle. See "What Is the WebLogic Diagnostic Framework Console Extension?" in Oracle Fusion Middleware Using the Diagnostic Framework Console Extension for Oracle WebLogic Server.

Using JMX to Monitor WebLogic Server

WebLogic Server provides its own set of MBeans that you can use to configure, monitor, and manage WebLogic Server resources. See "Understanding WebLogic Server MBeans" in Oracle Fusion Middleware Developing Custom Management Utilities With JMX for Oracle WebLogic Server.

Using WLST to Monitor WebLogic Server

The WebLogic Scripting Tool (WLST) is a command-line scripting interface that system administrators and operators use to monitor and manage WebLogic Server instances and domains. See "Understanding WebLogic Server MBeans" in Oracle Fusion Middleware Developing Custom Management Utilities With JMX for Oracle WebLogic Server.

Resources to Monitor WebLogic Server

The Oracle Technology Network at http://www.oracle.com/technology/index.html provides product downloads, articles, sample code, product documentation, tutorials, white papers, news groups, and other key content for WebLogic Server.

Third-Party Tools to Monitor WebLogic Server

Oracle partners with other companies that provide production monitoring and management tools. See Production Performance Management.

Filtering Loader Mechanism

WebLogic Server provides a filtering loader mechanism that allows the system classpath search to be bypassed when looking for specific application classes and resources that are on the application classpath. This mechanism requires a user configuration that specifies the specific classes and resources that bypass the system classpath search. See "Using a Filtering Classloader" in Oracle Fusion Middleware Developing Applications for Oracle WebLogic Server.

New for this release is the ability to filter resource loading requests. The basic configuration of resource filtering is specified in META-INF/weblogic-application.xml file and is similar to the class filtering. The the syntax for filtering resources is shown in the following example:

<prefer-application-resources>
<resource-name>x/y</resource-name>
<resource-name>z*</resource-name>
</prefer-application-resources>

In this example, resource filtering has been configured for the exact resource name "x/y" and for any resource whose name starts with "z". '*' is the only wild card pattern allowed. Resources with names matching these patterns are searched for only on the application classpath, the system classpath search is skipped.