# Configuration and User Guide

# Diagnostics and Troubleshooting

This section describes how to deal with problems that occur in both the server and the JVM and problems endemic to WLS-VE specifically. It also describes how to obtain information about your WLS-VE instance and provide that information to BEA Support.

This section includes information on the following subjects:

- Troubleshooting WLS-VE Problems
- Handling Suspend Files
- Displaying Version Information
- Reporting a Problem to BEA Support

## Troubleshooting WLS-VE Problems

This section provides information you will find helpful in solving problems that might occur with WLS-VE. Generally, you handle WebLogic Server and LiquidVM (the BEA JRockit component) problems the same way you would for their non-virtualized versions. You should follow BEA Support's instructions for information collection, augmented with those in Reporting a Problem to BEA Support. For BEA JRockit, you can use the standard tools available with BEA JRockit Mission Control—such as the JRockit Runtime Analyzer and Memory Leak Detector—to help you diagnose problems and collect relevant information about runtime activity.

This section contains information on the following subjects:

- Troubleshooting Common WLS-VE Issues
- Troubleshooting WLS Issues
- Troubleshooting LiquidVM Issues

### Troubleshooting Common WLS-VE Issues

Problems with WLS-VE not specifically associated with WLS or with LiquidVM, can probably be traced to configuration errors. This section will help you identify the problem and figure out what caused it and how to resolve it. If you cannot find the solution here, collect the necessary information about your system, as described in Reporting a Problem to BEA Support, and open a case with BEA Support.

Note: | If you have configured your installation using an NFS share, you may encounter certain error conditions related to the NFS configuration. Common NFS error conditions are described in the WLS-VE version 1.0 Installation and Configuration Guide.

The most common error conditions you might encounter are:

- "Could not find the disk" Error
- The Server Shuts Down Soon After Startup
- The Server Shuts Down Soon After Startup
- "netSend failed: -3" Error
- "Configured IP [...] in use by MAC" Error

#### "Could not find the disk" Error

Symptom: When you launch your instance and you get the following output in your OS console window:

    Starting WLS-MyServer. connect...configure...create...
    Could not find the disk: [storage2] wlsve/isoName.iso

Problem: When the virtual machine was created, VMware could not find the ISO image that you need to boot up WLS-VE.

Solution:

Check that you have uploaded the WLS-VE ISO image to the ESX server (see Copying the WLS-VE ISO Image in the WLS-VE Installation Guide for more information).

Verify that the `bea.lvm.info` file in your home directory points to the correct location on the ESX server. You can do this either by manually editing the `bea.lvm.info` file in your favorite editor or by rerunning the LiquidVM configuration wizard (`tools\virtualization\control_1.1\bin\lvm_configwizard.cmd` or `.sh`), as described in Configuring LiquidVM Connection Parameters.

Confirm that the ISO image exists, using the LiquidVM configuration wizard:

1. Select VM Host.
2. Select the Configuration tab and note the Datastore Name.
3. Select Browse Datastore and confirm that `wlsve922.iso` is available in your datastore.

#### The Server Shuts Down Soon After Startup

Symptom: The server shuts down soon after startup and a LiquidVM log file named `WLS-``<servername>``.lvm.out` appears in the domain directory on the local disk. In that file, you find the following:

    <Jan 9, 2008 6:50:23 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic
    Server 9.2 MP2  Fri Dec 7 01:21:28 EST 2007 1023546 >
    <Jan 9, 2008 6:50:30 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
    Enter username to boot WebLogic server:The application tried to read from keyboard (stdin). However,
    reading from keyboard is not possible when running LiquidVM.
    Please be aware that this could result in unexpected behaviour
    if the application really depends on keyboard input working
    <Jan 9, 2008 6:50:30 PM EST> <Error> <Security> <BEA-090783> <Server is
    Running in Development Mode and Native Library(terminalio) to read the
    password securely from commandline is not found.>
    <Jan 9, 2008 6:50:30 PM EST> <Notice> <WebLogicServer> <BEA-000388> <JVM
    called WLS shutdown hook. The server will force shutdown now>
    <Jan 9, 2008 6:50:30 PM EST> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>
    <Jan 9, 2008 6:50:30 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Problem: You didn't provide a user name and password either in the `security` directory (no `boot.properties` file) of the Administration Server's root directory or in the start script using the `WLS_USER` and `WLS_PW` properties. Keep in mind that WLS-VE does not support normal keyboard input, so you cannot enter a username and password on the keyboard.

Solution: Add a `boot.properties` file to the `security` directory of the Administration Server's root directory (see Creating a Boot Identity File for an Administration Server in the WLS 9.2 document, Managing Server Startup and Shutdown) or add a username and password in the start script and relaunch your WLS-VE server.

#### "netSend failed: -3" Error

Symptom: You receive the following error message on the Virtual Center console:

    000000 [rpcconn    WRN] netSend failed: -3
    000000 [rpcconn    WRN] Rpc call failed
    000000 [rpc        WRN] Rpc request failed: 3
    000000 [rpc        WRN] rpcDoReqeust returned 3
    000000 [rpc        WRN] rpcCall 3 returned 8549398

Problem: Your network configuration is incorrect.

Solution: In the start-up script, check your static IP address, your gateway, and your netmask and verify that they are correct. If they aren't, obtain the correct information and enter it in the respective property.

#### "Configured IP [...] in use by MAC" Error

Symptom: When you attempt to start a server, you receive this message:

    000000 [net   WRN] Configured IP [172.18.134.55] in use by MAC: 00:50:56:a0:
    06:96
    000001 [net   WRN] Network stack initialization FAILED: 98

Problem: Someone else is already using the IP address you have specified.

Solution: Another running VM might be using the same IP address. Do the following:

- Verify that you typed the IP address correctly.
- Verify that none of your running VMs already use that IP address.

If neither of the above is the case, someone else is using your IP address. Since finding out who that person is might be is difficult, contact your system administrator to obtain another IP address.

### Troubleshooting WLS Issues

WLS-VE can encounter the same type of server-related problems that can occur when running non-virtualized WebLogic Server. This section provides an overview of the kinds of WebLogic Server problems you should watch for when running WLS-VE. It includes information on these subjects:

- Performance Issues
- Server Failure
- Clustering Issues
- Other WLS Problems

#### Performance Issues

Often, a problem with WebLogic Server is the result of poor tuning. For example, pool sizes (such as pools for JDBC connections, Stateless Session EJBs, and MDBs) that don't maximize concurrency for the expected thread utilization can adversely affect performance. Similarly, applications that handle large amounts of data per request will experience a boost in performance if the chunk size—that is, a unit of memory that the WLS network layer uses to read data from and write data to sockets—on both the client and server sides can be increased, a process called tuning the chunk size.

You can find well-tested tuning and performance guidelines in WebLogic Server Performance and Tuning.

Specific tuning guidelines for WLS-VE are provided in Tuning the WLS-VE LiquidVM Kernel.

#### Server Failure

A server instance can fail and different events can lead to this failure. Often one failure condition leads to another. Loss of power, hardware malfunction, OS crashes, network partitions, and unexpected application behavior can all contribute to the failure of a server instance. Even in a clustered environment, server instances may fail periodically and you must be prepared for the recovery process. See Avoiding and Recovering From Server Failure in Managing Server Startup and Shutdown for information on dealing with server failure.

#### Clustering Issues

A number of cluster problems can affect the performance of WebLogic Server. These problems can occur for many reasons, including licensing and versioning errors, multicast addressing problems, errors or misspellings in start-up commands, and even a poorly-tuned memory management systems. You can find guidelines for troubleshooting cluster problems in Troubleshooting Common Problems in Using WebLogic Server Clusters.

#### Other WLS Problems

Other, non-specific problems can also occur with WLS. When these problems occur, they usually generate an error message with an associated error code. The Index of Messages by Message Range provides descriptions, possible causes, and corrective actions for all WebLogic Server error conditions.

### Troubleshooting LiquidVM Issues

Problems that don't originate with WLS may occur in LiquidVM and are typical to the kinds of problems you might encounter in non-virtualized JVMs. Problems such as these are documented in the BEA JRockit Diagnostics Guide (BEA JRockit is the JVM component of LiquidVM). This document provides information for either resolving the problem yourself or mining the necessary information required to open a case with BEA Support.

The types of LiquidVM problems you might encounter when running WLS-VE are:

- System crashes occur when the entire system shuts down involuntarily and usually without warning. See The System is Crashing.
- System freezes occur when the application stops answering requests but the process is still there. See BEA JRockit is Freezing.
- Slow startups usually occur when BEA JRockit's optimizing compiler must run extensively to ensure that the most efficient code possible is compiled. See BEA JRockit Starts Slowly.
- Poor performance usually occurs when your application experiences poor throughput. This usually indicates that the memory management system has not been tuned for optimal performance. See Low Overall Throughput.
- Occasional slow response times usually indicate that transactions are taking too long to execute, a bottleneck most often caused by garbage collection pause times lasting too long. See Long Latencies.
- Performance degrading after the application has been running is characterized by your application's behavior: it may be working fine early in its run, but after a while it may report the wrong results, or throw exceptions where it shouldn't, or it simply crashes or hangs at roughly the same time each time you run it. See BEA JRockit's Performance Degrades Over Time.

Note that in most UNIX OSs there is a file descriptor limit that limits the number of files and sockets you can have open. LiquidVM does not have such limits so there is no need (and no way) to set a file descriptor limit.

For complete information on BEA JRockit problem determination and resolution, see BEA JRockit Tools in the BEA JRockit Diagnostics Guide.

## Handling Suspend Files

When WLS-VE crashes, the VM goes into a state of suspension. A pause button will appear on the VirtualCenter and information about the crash will be written to the console. When a suspend file is created, do the following:

1. `tar-gzip` the suspend file. You will find it on the VM's home directory on the ESX server; it will have a filetype of `.vmss`.
2. Copy the `tgz` file from the ESX server to your normal environment (for example, your `My Documents/` folder).
3. Upload the `tgz` file to BEA Support.

Be aware that you might not realize that your machine has actually crashed when it suspends. You should avoid the temptation to resume execution, as you might lose critical information that would be helpful in diagnosing the problems causing the crash. You should also be aware that suspend files are huge and might not be easy for you to copy from the ESX server.

## Displaying Version Information

A critical piece of information that Support will need to help diagnose any problems you report to them is the version number. You can find this number in the file `LVM_VERSION`, which is located in the `BEA_HOME`/`tools/virtualization/control_1.1/` directory.

Open this file to find the version number; for example:

LiquidVM R1.1.7.0-93732

## Reporting a Problem to BEA Support

If you determine that you need to file a trouble report, this section discusses what you need to do before opening the case to ensure that you supply the support personnel assigned to your issue as complete picture of what is wrong as possible. The more information you can provide, the more quickly will the support staff be able to resolve your issue. This section includes information on these subjects:

- Trouble Reporting Process Overview
- Identify Your Problem Type
- Verify That You're Running a Supported Configuration
- Collect Enough Information to Define Your Issue

### Trouble Reporting Process Overview

When you encounter a problem with WebLogic Server Virtual Edition and can't resolve it using the information provided in the relevant BEA documentation, you need to collect the information that best describes your problem and open a case with BEA Support. If you have a service agreement with BEA, the normal process is to contact your Level 1 service provider, who will make the initial attempts to correct the problem. If the case cannot be solved by the Level 1 staff, it is escalated to the Level 2 staff, who will draw on their particular expertise to get your JVM running again. For serious problems, the issue will be handled by the Level 3 staff (the WebLogic Server Virtual Edition developers).

### Identify Your Problem Type

Is your machine crashing? Is it running slowly or returning unpredictable results? These are the kind of symptoms that indicate a problem with WebLogic Server Virtual Edition. Being able to identify what kind of problem you are experiencing will help you know what kind of information you need to include when you open the trouble report.

### Verify That You're Running a Supported Configuration

Before submitting a bug, verify that the environment in which the problem was found is a supported configuration. See the Supported Configurations page for WLS-VE 1.1.

### Collect Enough Information to Define Your Issue

In addition to testing with the latest update release, use the following guidelines to prepare for submitting a trouble report:

1. Collect as much relevant data as possible. For example, generate a thread-dump in the case of a deadlock, or locate the core file (where applicable) and `hs_err` file in the case of a crash. In all cases it is important to document the environment and the actions performed just before the problem is encountered.
2. Where applicable, try to restore the original state and reproduce the problem using the documented steps. This helps to determine if the problem is reproducible or an intermittent issue.
3. If the issue is reproducible, try to narrow the problem. In some cases, a bug can be demonstrated with a small standalone test case. Bugs demonstrated by small test cases will typically be easy to diagnose when compared to test cases that consists of a large complex application.
4. Search the bug database to see if the bug, or similar bugs, have been reported. If the bug has already been reported, the bug report may have further information. For example, if the bug has already been fixed, it will indicate the release that the bug was fixed in. The bug may also contain information, such as a workaround or include comments in the evaluation that explain, in further detail, the circumstances that cause the bug to arise.

If you conclude that the bug has not already been reported, then it is important to submit a new bug.