Sun Java System Application Server Platform Edition 8.1 2005Q2 Update 2 Troubleshooting Guide

- Previous: Preface
- Next: Chapter 2 Common Problems

# Chapter 1 Overview

This chapter provides a description of the tools, methods, and information sources available for troubleshooting the Sun Java System Application Server Server 8.1 Platform Edition. Guidelines for evaluating and investigating a problem are included.

- Planning Ahead

- Identifying the Problem

- Seeking a Solution

# Planning Ahead

As applications get deployed, undeployed, and redeployed, and as you experiment with different server configuration settings, there may be times when your server gets into a confused or unstable state. In such cases, it is useful to have a previously saved working configuration on which to fall back. This is not problem solving, per se, but rather a way to avoid problems in the first place.

The Application Server `asadmin` command includes a backup-domain option that backs up the domain(s) you specify. Use this option to take periodic “snapshots” of your server configuration. Then, if necessary, use the restore-domain option to restore one or more domains to a known working state.

Refer to the Application Server Administration Guide for complete instructions on using the `asadmin backup-domain` and `restore-domain` options. Briefly, however, for the purposes of this Troubleshooting Guide, use the following procedure to backup and restore a server configuration:

## To back up and restore a server configuration

##### Steps

1. Start the Application Server.

       install_dir/bin/asadmin start-domain domain_name

2. Stop the domain.

       install_dir/bin/asadmin stop-domain domain_name

3. Back up the domain.

       install_dir/bin/asadmin backup-domain domain_name

   Backed up directories are stored by default in the install_dir`/backups` directory.

4. Make changes to the Application Server configuration and/or domain(s), as desired.

5. If necessary, restore the server and/or domain configuration to the state saved in Step 3, above.

       install_dir/bin/asadmin restore-domain --filename backup_file domain_name

# Identifying the Problem

J2EE application servers are typically deployed in complex and highly sophisticated operating environments. The Sun Java System Application Server covers a broad range of technologies, including Java, Java servlets, XML, JSP, JDBC data sources, EJB technology, and more. Other products and tools associated with the Application Server are LDAP, Web Server, SunONE Message Queue, deployment and migration tools, and so on. Understanding and diagnosing complex issues involving so many disparate components requires thorough knowledge and a careful diagnostic process.

Gathering any or all of the following information will make it easier to classify a problem and search for solutions. Note that operating system utilities, such as `pkginfo` and `showrev` on Solaris and `rpm` on Linux, are helpful in gathering system information.

## Verifying configuration information

##### Steps

1. What are the exact version numbers of the operating system and products installed?

2. Have any patches been applied? If so, specify product and operating system patch numbers.

3. How is the system configured?

4. What system resources does the system have (memory, disk, swap space, and so on)?

5. How many application servers, web servers, and directory servers are installed?

6. How is the web server connected to Application Server? On the same machine or not?

7. How is the Application Server connected to the directory server?

8. Are application servers in a cluster or not?

9. Was any upgrade done? If so, what were source and target versions?

10. Was a migration done? If so, what were source and target versions?

11. Have any new applications been deployed?

12. Is SSL enabled or not?

13. What versions of the HADB and the backend database are being used?

14. What JDBC driver is being used to access the database?

15. What JDK version is being used?

16. What are the JVM heap, stack, and garbage collection-related parameters set to?

17. What are the JVM options?

18. What third-party technologies are being used in the installation?

19. Are the interoperating component versions in compliance with the compatibility matrix specified in the release notes?

    After gathering this information:

    - Collect web server error and access log data (web server instance-specific).

    - Collect any Application Server stack traces. Note that a fresh set of logs associated with the specific problem should be run. This avoids scanning gigabytes of irrelevant log information.

    - Determine the sequence of events that occurred when the problem first appeared, including any steps that may already have been taken to resolve the problem.

# Seeking a Solution

After identifying the problem and formulating a preliminary hypothesis of what may be wrong, you are ready to do someinvestigation.

The following topics are addressed in this section:

- Verify System Configuration

- Evaluate Messages

- Examine Log Files

- See if the Problem has been Solved Before

- Search the Product Documentation

- Search the Knowledge Base

- Search or Participate in the Online Forum

- Contact Support

## Verify System Configuration

Sometimes the most obvious solutions are overlooked, and so the first step is to verify the system configuration. Refer to the Sun Java System Application Server 8.1 Release Notes for the most up-to-date system requirements and dependencies.

## Evaluate Messages

Messages generally include information about the attempted action, the outcome of the action, and, if applicable, the cause of jeopardy or failure.

### Types of Messages

The log files contain the following general types of message entries:

- Error – These messages mark critical failures that cause status to be reported as Failed. Error messages generally provide detailed information about the nature and the cause of the problem that occurred.

- Warning – These messages mark non-critical failures. Warning messages generally contain information about the cause and the nature of the failure, and also provide possible remedies.

- Information – These messages mark normal completion of particular tasks.

### Error Messages

A problem is often accompanied by an error message that prevents the application from proceeding.

- In some cases, the message is very clear about what is wrong and what needs to be done, if anything, to fix it. For example, if you start a domain using the `asadmin start-domain` command, then inadvertently issue the same command again after the domain has started, the following message is displayed:

      userD:\\Sun\\studio5_se\\appserver8\\bin\>asadmin start-domain
      Domain already started : domain1
      Domain domain1 Started.

  In this case, the message gives clear guidance and the problem can be disregarded.

- Sometimes an error message gives only general information about the problem or solution, or suggests multiple possibilities. For example:

      [16/Jun/2003:22:20:50] SEVERE ( 2204): WEB0200: Configuration error
      in  web module [JAXBProjectStudio] (while initializing virtual server
      [server1]) com.iplanet.ias.config.ConfigException: Failed to load
      deployment descriptor for: JAXBProjectStudio cause:
      java.io.FileNotFoundException:

  In this case, the problem is not obvious, or there might be multiple things wrong. You might have to consider various possibilities and perhaps a number of solutions. If the proposed fix is time consuming or costly, take steps to ensure that the fix is likely to be correct before actually doing anything.

- Some error messages are either not helpful or provide little guidance; for example:

      [23/Jun/2003:16:50:45] WARNING ( 1972): for host 127.0.0.1 trying to GET
      /SupplierServiceClient1/SupplierServiceClient1_SOAP.html, send-file
      reports:  HTTP4144: error sending D:/Sun/studio5_se/appserver8/domains/
      domain1/server1/applications/j2ee-modules/SupplierServiceClient1_1/
      SupplierServiceClient1_SOAP.html (Overlapped I/O operation is in progress.)
      status=1:5

  In this case, there is very little information to go on. It is especially important to identify the exact situation that caused the error, and what the symptoms are before proceeding.

  For descriptions of all the Application Server error messages, refer to the Sun Java System Application Server Error Message Reference at the following location:

  `http://docs.sun.com/db/coll/ApplicationServer8_04q4`

## Examine Log Files

A number of the Application Server subsystems create log files and log their events to these files. The primary purpose of these log files is to provide troubleshooting information.

In addition to the message text, a logged message provides the following information:

- Date and time of the event

- Log level for the event — Application Server-specified log level ID or name

- Process identifier (PID) — PID of the Application Server process

- (optional) Virtual server identifier (VSID) — VSID that generated the message

- Message identifier (MID) — subsystem and a four digit integer

- Message data

  The specific logs associated with each Application Server problem area are discussed in the associated chapters of this manual.

### Log Levels

The Application Server has many log levels that can be set in the Administration GUI (`FINEST`, `FINER`, `FINE`, `CONFIG`, `INFO`, `WARNING`, `SEVERE`, `ALERT`, and `FATAL`). All messages are logged when the log level is set to `FINEST` and only serious error messages appear if the log level is set to `FATAL`.

Note that the more detailed log levels (`FINEST`, `FINER`, `FINE`) can generate high volumes of log information for certain events, which may make it appear at first glance that there is an error condition when in fact there is not.

All messages with a log level less than the default level of `INFO` (`FINEST`, `FINER`, `FINE`, and `CONFIG`) provide information related to debugging and must be specifically enabled. Instructions for doing this are contained in the Sun Java System Application Server Administrator's Guide.

In addition to the standard JDK log levels, the Application Server has added log levels designed to map more intuitively to the Application Server log file (`server.log`) and to tightly integrate with Solaris. The log levels `ALERT` and `FATAL` are specific to the Application Server and are not implemented in the JDK1.4 logging API.

Note –

For information on the event log mechanism used in the Microsoft Windows operating environment, refer to the Windows help system index using the keywords Event Logging. If you choose to send logs to the Windows `server.log` file, only messages with a log level of INFO, WARNING, SEVERE, ALERT, or FATAL are logged to the Windows Event Log.

### Log Options

The Administration GUI provides the following two logging options:

- Option 1 — Log `stdout` (`System.out.print`) content to the event log

- Option 2 — Log `stderr` (`System.err.print`) content to the event log

  When these options are set, `stdout` and `stderr` messages are written to the `server.log` file. (The event log is a `syslog` daemon on Solaris and Event Log on Microsoft Windows.)

  If the above options are not set:

- Anything written to `stdout` or `stderr` (that is, using `System.out` or `System.err`) will not appear in the logs.

- Messages logged with the JDK logger will appear in the logs.

- Messages written to `stdout` or `stderr` appear with the `INFO` level, but do not have a message ID.

### Client Side Logging

The Application Client Container (ACC) has its own log service and can only log to a local file. The ACC typically runs in its own process, on a different host from the Application Server. It has its own logging infrastructure and its own log file. The `sun-acc.xml` file contains the ACC configuration. Refer to the Sun Java System Application Server Application Server Developer's Guide to Clients for more information.

### Obtaining a Thread Dump

This section explains how to obtain a thread dump for Application Server 8.1. By default, the server dumps a core file and restarts with the `-Xrs java-option` flag in the `server.xml` file.

#### On UNIX

The following procedure describes how to obtain a server thread dump on UNIX.

#### To obtain a server thread dump on UNIX

##### Steps

1. Verify that the `server.xml` file for the affected server instance does not include the `-Xrs java-option` flag. Remove the `-Xrs java-option` flag if it exists.

2. If the option is changed, restart the server instance.

3. Use the `ps` command to determine the `java` and/or `appservDAS` processes under which the application server is running.

4. Run the following command on the application server instance:

       kill -3 pid

   The `kill` command redirects the thread dump to the `server.log` file for server the instance.

#### On Windows

The following procedure describes how to obtain a server thread dump on Windows.

#### To obtain a server thread dump on Windows

##### Steps

1. Verify that the `server.xml` file for your server instance does not include the `-Xrs` `java-option` flag. Remove the `-Xrs` `java-option` flag if it exists.

2. If the option was changed, restart your Application Server.

3. Type `ctrl-brk` in the Application Server window. The thread dump will be redirected to the `server.log` file for the instance.

## See if the Problem has been Solved Before

A good initial step is to scan this Troubleshooting Guide to see if the problem is addressed here. If so, select the appropriate solution. Many of the solutions contain references to other documents in the Application Server document collection for additional details, explanations, or examples.

## Search the Product Documentation

Start by reading the Release Notes for the version of the product you are troubleshooting.

The documentation for this Application Server product release is available at:

`http://docs.sun.com/db/coll/ApplicationServer81`

Descriptions of the Application Server manuals are listed in Application Server Documentation Set.

## Search the Knowledge Base

The Knowledge Base is a collection of articles on product issues that provide information helpful for troubleshooting. To access the Knowledge Base:

### To search the Knowledge Base

##### Steps

1. Go to SunSolve.

2. Under SunSolve Collections, click the Search Collections link.

3. Select the checkbox for the collection(s) to search.

4. Click Next.

5. Enter the search criteria.

6. Click Go.

## Search or Participate in the Online Forum

Browse directly in any of the online forums, or log in and register to start posting messages. The Application Server online forum is available at: `http://swforum.sun.com/jive/index.jsp?cat=7`

## Contact Support

When necessary, gather together the information you have acquired and contact technical support at `http://www.sun.com/service/contacting`.

- Previous: Preface
- Next: Chapter 2 Common Problems

- © 2010, Oracle Corporation and/or its affiliates