Sun Java System Application Server Enterprise Edition 8.1 2005Q2 Performance Tuning Guide

- Previous: Transaction Service Settings
- Next: ORB Settings

# HTTP Service Settings

Monitoring and tuning the HTTP server instances that handle client requests are important parts of ensuring peak Application Server performance.

- Monitoring the HTTP Service

- Tuning the HTTP Service

- Tuning HTTP Listener Settings

- Migrating From Version 7

## Monitoring the HTTP Service

Enable monitoring statistics for the HTTP service using either Admin Console or `asadmin`. In the Admin Console, the monitoring level (LOW or HIGH) has no effect on monitoring the HTTP Service.

With `asadmin`, use the following command to list the monitoring parameters available:

    list --user admin --port 4848
     -m server-instance-name.http-service.*

where server-instance-name is the name of the server instance.

Use the following command to get the values:

    get --user admin --port 4848 -m server.http-service.parameter-name.*

where parameter-name is the name of the parameter to monitor.

Statistics collection is enabled by default. Disable it by adding the following property to `domain.xml` and restart the server:

    <property name="statsProfilingEnabled" value="false" />

Disabling statistics collection will increase performance.

You can also view monitoring statistics with the Admin Console. The information is divided into the following categories:

- General HTTP Statistics (http-service)

- DNS Cache Information (dns)

- Connection Queue

- File Cache Information (file-cache)

- Keep Alive (keep-alive)

- Thread Pool (pwc-thread-pool)

### General HTTP Statistics (http-service)

The Admin Console provides the following performance-related HTTP statistics:

- Average load for last minute

- Is VirtualServer Overflow enabled?

- HttpServer Version

- HttpServer ID

- Rate at which bytes are being received

- Maximum amount of threads

- HttpServer Time Started

- Maximum amount of virtual servers

- Is profiling enabled?

- Time in seconds HttpService has been running

- Average load for last 15 minutes

- Average load for last 5 minutes

- Rate at which bytes are being transmitted

### DNS Cache Information (dns)

The DNS cache caches IP addresses and DNS names. Your server’s DNS cache is disabled by default. In the DNS Statistics for Process ID All page under Monitor in the web-based Administration interface the following statistics are displayed:

#### Enabled

If the DNS cache is disabled, the rest of this section is not displayed.

By default, the DNS cache is off. Enable DNS caching with the Admin Console by setting the DNS value to “Perform DNS lookups on clients accessing the server”.

#### CacheEntries (CurrentCacheEntries / MaxCacheEntries)

The number of current cache entries and the maximum number of cache entries. A single cache entry represents a single IP address or DNS name lookup. Make the cache as large as the maximum number of clients that access your web site concurrently. Note that setting the cache size too high is a waste of memory and degrades performance.

Set the maximum size of the DNS cache by entering or changing the value in the Size of DNS Cache field of the Performance Tuning page.

#### HitRatio

The hit ratio is the number of cache hits divided by the number of cache lookups.

This setting is not tunable.

Note –

If you turn off DNS lookups on your server, host name restrictions will not work and IP addresses will appear instead of host names in log files.

#### Caching DNS Entries

It is possible to also specify whether to cache the DNS entries. If you enable the DNS cache, the server can store hostname information after receiving it. If the server needs information about the client in the future, the information is cached and available without further querying. specify the size of the DNS cache and an expiration time for DNS cache entries. The DNS cache can contain 32 to 32768 entries; the default value is 1024. Values for the time it takes for a cache entry to expire can range from 1 second to 1 year specified in seconds; the default value is 1200 seconds (20 minutes).

#### Limit DNS Lookups to Asynchronous

Do not use DNS lookups in server processes because they are resource-intensive. If you must include DNS lookups, make them asynchronous.

#### Enabled

If asynchronous DNS is disabled, the rest of this section will not be displayed.

#### NameLookups

The number of name lookups (DNS name to IP address) that have been done since the server was started. This setting is not tunable.

#### AddrLookups

The number of address loops (IP address to DNS name) that have been done since the server was started. This setting is not tunable.

#### LookupsInProgress

The current number of lookups in progress.

### Connection Queue

- Total Connections Queued: Total connections queued is the total number of times a connection has been queued. This includes newly accepted connections and connections from the keep-alive system.

- Average Queuing Delay: Average queueing delay is the average amount of time a connection spends in the connection queue. This represents the delay between when a request connection is accepted by the server, and a request processing thread (also known as a session) begins servicing the request.

### File Cache Information (file-cache)

The file cache caches static content so that the server handles requests for static content quickly. The file-cache section provides statistics on how your file cache is being used.

For information on tuning the file cache, see HTTP File Cache.

- Number of Hits on Cached File Content

- Number of Cache Entries

- Number of Hits on Cached File Info

- Heap Space Used for Cache

- Number of Misses on Cached File Content

- Cache Lookup Misses

- Number of Misses on Cached File Content

- Max Age of a Cache Entry: The maximum age displays the maximum age of a valid cache entry.

- Max Number of Cache Entries

- Max Number of Open Entries

- Is File Cached Enabled?: If the cache is disabled, the other statistics are not displayed. The cache is enabled by default.

- Maximum Memory Map to be Used for Cache

- Memory Map Used for cache

- Cache Lookup Hits

- Open Cache Entries: The number of current cache entries and the maximum number of cache entries are both displayed. A single cache entry represents a single URI. This is a tunable setting.

- Maximum Heap Space to be Used for Cache

### Keep Alive (keep-alive)

The Admin Console provides the following performance-related keep-alive statistics:

- Connections Terminated Due to ClientConnection Timed Out

- Max Connection Allowed in Keep-alive

- Number of Hits

- Connections in Keep-alive Mode

- Connections not Handed to Keep-alive Thread Due to too Many Persistent Connections

- The Time in Seconds Before Idle Connections are Closed

- Connections Closed Due to Max Keep-alive Being Exceeded

### Thread Pool (pwc-thread-pool)

The Admin Console provides the following thread pool statistics:

- Idle/Peak/Limit: Idle indicates the number of threads that are currently idle. Peak indicates the peak number in the pool. Limit indicates the maximum number of native threads allowed in the thread pool, and is determined by the setting of `NativePoolMaxThreads`.

- Work Queue Length /Peak /Limit: These numbers refer to a queue of server requests that are waiting for the use of a native thread from the pool.

  - The Work Queue Length is the current number of requests waiting for a native thread.

  - Peak is the highest number of requests that were ever queued up simultaneously for the use of a native thread since the server was started. This value can be viewed as the maximum concurrency for requests requiring a native thread.

  - Limit is the maximum number of requests that can be queued at one time to wait for a native thread, and is determined by the setting of `NativePoolQueueSize`.

## Tuning the HTTP Service

The settings for the HTTP service are divided into the following categories in the Admin Console:

- Access Log

- Request Processing

- Keep Alive

- Connection Pool

- HTTP Protocol

- HTTP File Cache

### Access Log

Disable access logging when performing benchmarking. Access Logging is enabled by default. To disable it, in HTTP Service click Add Property, and add the following property:

- name: `accessLoggingEnabled`

- value: false

You can set the following access log properties:

- Rotation (enabled/disabled). Enable rotation to ensure that the logs don’t run out of disk space.

- Rotation Policy:ime-based or size-based. Size-based is the default.

- Rotation Interval.

### Request Processing

On the Request Processing tab of the HTTP Service page, tune the following HTTP request processing settings:

- Thread Count

- Initial Thread Count

- Request Timeout

- Buffer Length

#### Thread Count

The Thread Count parameter specifies the maximum number of simultaneous requests the server can handle. The default value is 128. When the server has reached the limit or request threads, it defers processing new requests until the number of active requests drops below the maximum amount. Increasing this value will reduce HTTP response latency times.

In practice, clients frequently connect to the server and then do not complete their requests. In these cases, the server waits a length of time specified by the Request Timeout parameter.

Also, some sites do heavyweight transactions that take minutes to complete. Both of these factors add to the maximum simultaneous requests that are required. If your site is processing many requests that take many seconds, you might need to increase the number of maximum simultaneous requests.

Adjust the thread count value based on your load and the length of time for an average request. In general, increase this number if you have idle CPU time and requests that are pending; decrease it if the CPU becomes overloaded. If you have many HTTP 1.0 clients (or HTTP 1.1 clients that disconnect frequently), adjust the timeout value to reduce the time a connection is kept open.

Suitable Request Thread Count values range from 100 to 500, depending on the load. If your system has extra CPU cycles, keep incrementally increasing thread count and monitor performance after each incremental increase. When performance saturates (stops improving), then stop increasing thread count.

#### Initial Thread Count

The Initial Thread Count property specifies the minimum number of threads the server initiates upon start-up. The default value is 48. Initial Thread Count represents a hard limit for the maximum number of active threads that can run simultaneously, which can become a bottleneck for performance.

#### Request Timeout

The Request Timeout property specifies the number of seconds the server waits between accepting a connection to a client and receiving information from it. The default setting is 30 seconds. Under most circumstances, changing this setting is unnecessary. By setting it to less than the default 30 seconds, it is possible to free up threads sooner. However, disconnecting users with slower connections also helps.

#### Buffer Length

The size (in bytes) of the buffer used by each of the request processing threads for reading the request data from the client.

Adjust the value based on the actual request size and observe the impact on performance. In most cases the default should suffice. If the request size is large, increase this parameter.

### Keep Alive

Both HTTP 1.0 and HTTP 1.1 support the ability to send multiple requests across a single HTTP session. A server can receive hundreds of new HTTP requests per second. If every request was allowed to keep the connection open indefinitely, the server could become overloaded with connections. On Unix/Linux systems, this could easily lead to a file table overflow.

The Application Server’s Keep Alive system addresses this problem. A waiting keep alive connection has completed processing the previous request, and is waiting for a new request to arrive on the same connection. The server maintains a counter for the maximum number of waiting keep-alive connections. If the server has more than the maximum waiting connections open when a new connection waits for a keep-alive request, the server closes the oldest connection. This algorithm limits the number of open waiting keep-alive connections.

If your system has extra CPU cycles, incrementally increase the keep alive settings and monitor performance after each increase. When performance saturates (stops improving), then stop increasing the settings.

The following HTTP keep alive settings affect performance:

- Thread Count

- Max Connections

- Time Out

- Keep Alive Query Mean Time

- Keep Alive Query Max Sleep Time

#### Thread Count

Thread Count determines the number of threads in the Keep Alive subsystem. Adjust this setting to be a small multiple of the number of processors on the system. For example, a two-CPU system can have two or four keep-alive threads.

The default is one. Do not change the default for a server with a small number of users and Max Connections.

#### Max Connections

Max Connections controls the maximum number of keep-alive connections the server maintains. The possible range is zero to 32768, and the default is 256.

Adjust this setting based on number of keep alive connections the server is expected to service and the server’s load, because it will add up to resource utilization and might increase latency.

The number of connections specified by Max Connections is divided equally among the keep alive threads. If Max Connections is not equally divisible by Thread Count, the server can allow slightly more than Max Connections simultaneous keep alive connections.

#### Time Out

Time Out determines the maximum time (in seconds) that the server holds open an HTTP keep alive connection. A client can keep a connection to the server open so that multiple requests to one server can be serviced by a single network connection. Since the number of open connections that the server can handle is limited, a high number of open connections will prevent new clients from connecting.

The default time out value is 30 seconds. Thus, by default, the server will close the connection if idle for more than 30 seconds. The maximum value for this parameter is 300 seconds (5 minutes).

The proper value for this parameter depends upon how much time is expected to elapse between requests from a given client. For example, if clients are expected to make requests frequently then, set the parameter to a high value; likewise, if clients are expected to make requests rarely, then set it to a low value.

#### Keep Alive Query Mean Time

Keep Alive Query Mean Time specifies the interval between polling keep alive connections. If this parameter has a value of n milliseconds, the response time seen by a client that has requested a keep alive connection will have an overhead between 0 and n milliseconds.

The default value of this parameter is one millisecond, which works well for an expected concurrent load of less than 300 keep alive connections. The default value can severely reduce the scalability with higher concurrent loads. For applications with higher connection loads, increase the default value.

Set this parameter with `asadmin` or in Admin Console HTTP Service page, by choosing Add Property and specifying:

- Name: `keep-alive-query-mean-time`

- Value: number of milliseconds

#### Keep Alive Query Max Sleep Time

Keep Alive Query Max Sleep Time specifies the maximum time (in milliseconds) to wait that after polling keep alive connections for further requests. If your system has extra CPU cycles, keep incrementally increasing this parameter and monitor performance after each incremental increase. When performance saturates (stops improving), then stop increasing the settings.

Set this parameter with `asadmin` or in the Admin Console HTTP Service page, by choosing Add Property and specifying:

- Name: `keep-alive-query-max-sleep-time`

- Value: number of milliseconds

### Connection Pool

Connection queue information shows the number of sessions in the queue, and the average delay before the connection is accepted.

If your system has extra CPU cycles, keep incrementally increasing connection pool settings and monitor performance after each incremental increase. When performance saturates (stops improving), then stop increasing the settings.

Connection pool settings that affect performance are:

- Max Pending Count

- Queue Size

#### Max Pending Count

Max Pending Count specifies the maximum number of pending connections on the listen socket. Adjust Max Pending Count only when there is a heavy load on the system. For low to medium loads, the default will be acceptable.

After observing system behavior, change the value accordingly, otherwise the server will start dropping connections. Connections that time out on a listen socket whose backlog queue is full will fail. If Max Pending Count is close to the limit, increase the maximum connection queue size to avoid dropping connections under heavy load.

#### Queue Size

Queue Size specifies the number of outstanding (yet to be serviced) connections that the server can have. For heavily loaded systems (with many users) that have limited request processing threads, adjust this setting to a higher value.

Note –

Setting the connection queue size too high can degrade server performance. It was designed to prevent the server from becoming overloaded with connections it cannot handle. If the server is overloaded, increasing the connection queue size will increase the latency of request handling, and the connection queue will fill up again.

#### Send Buffer Size

Specifies the size (in bytes) of the send buffer used by sockets.

#### Receive Buffer Size

Specifies the size (in bytes) of the receive buffer used by sockets.

The Send Buffer Size and Receive Buffer Size are the buffer sizes allocated for output and input buffers, respectively. To tune these parameters, increase them methodically and observe the impact on performance. Stop increasing the values when performance saturates (does not increase significantly).

### HTTP Protocol

The only HTTP Protocol attribute that significantly affects performance is DNS Lookup Enabled.

#### DNS Lookup Enabled

This setting specifies whether the server performs DNS (domain name service) lookups on clients that access the server. When DNS lookup is not enabled, when a client connects, the server knows the client’s IP address but not its host name (for example, it knows the client as 198.95.251.30, rather than `www.xyz.com`). When DS lookup is enabled, the server will resolve the client’s IP address into a host name for operations like access control, common gateway interface (CGI) programs, error reporting, and access logging.

If the server responds to many requests per day, reduce the load on the DNS or NIS (Network Information System) server by disabling DNS lookup. Enabling DNS lookup will increase the latency and load on the system—do so with caution.

### HTTP File Cache

The Application Server uses a file cache to serve static information faster. The file cache contains information about static files such as HTML, CSS, image, or text files. Enabling the HTTP file cache will improve performance of applications that contain static files.

Set the file cache attributes in the Admin Console under Configurations > config-name > HTTP Service (HTTP File Cache).

#### Max Files Count

Max Files Count determines how many files are in the cache. If the value is too big, the server caches little-needed files, which wastes memory. If the value is too small, the benefit of caching is lost. Try different values of this attribute to find the optimal solution for specific applications—generally, the effects will not be great.

#### Hash Init Size

Hash Init Size affects memory use and search time, but rarely will have a measurable effect on performance.

#### Max Age

This parameter controls how long cached information is used after a file has been cached. An entry older than the maximum age is replaced by a new entry for the same file.

If your web site’s content changes infrequently, increase this value for improved performance. Set the maximum age by entering or changing the value in the Maximum Age field of the File Cache Configuration page in the web-based Admin Console for the HTTP server node and selecting the File Caching Tab.

Set the maximum age based on whether the content is updated (existing files are modified) on a regular schedule or not. For example, if content is updated four times a day at regular intervals, you could set the maximum age to 21600 seconds (6 hours). Otherwise, consider setting the maximum age to the longest time you are willing to serve the previous version of a content file after the file has been modified.

#### Small/Medium File Size and File Size Limit

The cache treats small, medium, and large files differently. The contents of medium files are cached by mapping the file into virtual memory (Unix/Linux platforms). The contents of small files are cached by allocating heap space and reading the file into it. The contents of large files are not cached, although information about large files is cached.

The advantage of distinguishing between small files and medium files is to avoid wasting part of many pages of virtual memory when there are lots of small files. So the Small File Size Limit is typically a slightly lower value than the VM page size.

#### File Transmission

When File Transmission is enabled, the server caches open file descriptors for files in the file cache, rather than the file contents. Also, the distinction normally made between small, medium, and large files no longer applies since only the open file descriptor is being cached.

By default, File Transmission is enabled on Windows, and disabled on UNIX. On UNIX, only enable File Transmission for platforms that have the requisite native OS support: HP-UX and AIX. Don’t enable it for other UNIX/Linux platforms.

## Tuning HTTP Listener Settings

Change HTTP listener settings in the Admin Console under Configurations > config-name > HTTP Service > HTTP Listeners > listener-name.

### Network Address

For machines with only one network interface card (NIC), set the network address to the IP address of the machine (for example, 192.18.80.23 instead of default 0.0.0.0). If you specify an IP address other than 0.0.0.0, the server will make one less system call per connection. Specify an IP address other than 0.0.0.0 for best possible performance. If the server has multiple NIC cards then create multiple listeners for each NIC.

### Acceptor Threads

The Acceptor Threads setting specifies how many threads you want in accept mode on a listen socket at any time. It is a good practice to set this to less than or equal to the number of CPUs in your system.

In the Application Server, acceptor threads on an HTTP Listener accept connections and put them onto a connection queue. Session threads then pick up connections from the queue and service the requests. The server posts more session threads if required at the end of the request.

The policy for adding new threads is based on the connection queue state:

- Each time a new connection is returned, the number of connections waiting in the queue (the backlog of connections) is compared to the number of session threads already created. If it is greater than the number of threads, more threads are scheduled to be added the next time a request completes.

- The previous backlog is tracked, so that n threads are added (n is the HTTP Service’s Thread Increment parameter) until one of the following is true:

  - The number of threads increases over time.

  - The increase is greater than n.

  - The number of session threads minus the backlog is less than n.

  To avoid creating too many threads when the backlog increases suddenly (such as the startup of benchmark loads), the server makes the decision whether more threads are needed only once every 16 or 32 connections, based on how many session threads already exist.

## Migrating From Version 7

If you are migrating an existing installation from Application Server version 7.x to version 8, consult the following table to see the mapping of tunable parameters.

Table 3–3 Mapping of tunable settings from version 7 to version 8

Tunable Setting in version 7.x | Tunable Setting in version 8.1
--- | ---
RqThrottle | Thread Count (thread-count)
RqThrottleMin | Initial Thread Count (initial-thread-count)
ConnQueueSize | Queue Size (queue-size-in-bytes)
KeepAliveThreads | Thread Count (keep-alive-thread-count)
KeepAliveTimeout | Time Out (timeout-in-seconds)
MaxKeepAliveConnections | Max Connections (max-connections)
KeepAliveQueryMeanTime | keep-alive-query-mean-time
KeepAliveQueryMaxSleepTime | keep-alive-query-max-sleep-time
ListenQ | Thread Count (max-pending-count)
AcceptTimeout | Request Time Out
HeaderBufferSize | Buffer Length

- Previous: Transaction Service Settings
- Next: ORB Settings

- © 2010, Oracle Corporation and/or its affiliates