# Configuration and User Guide

# Diagnostics and Troubleshooting

This section describes how to deal with issues that might occur in both WLS and the JVM, as well as issues that are specific WLS-VE. It also describes how to obtain information about your WLS-VE instance and provide that information to BEA Support.

This section includes information on the following subjects:

- Troubleshooting WLS-VE
- Handling Suspend Files
- Displaying Version Information
- Reporting a Problem to BEA Support

## Troubleshooting WLS-VE

This section provides information you will find helpful in solving issues that might occur with WLS-VE. Generally, you handle WLS and LiquidVM (the BEA JRockit component) issues the same way you would for their non-virtualized versions. You should follow BEA Support’s instructions for information collection, augmented with those in Reporting a Problem to BEA Support. For BEA JRockit, you can use the standard tools available with BEA JRockit Mission Control, such as the JRockit Runtime Analyzer and Memory Leak Detector, to help you diagnose issues and collect relevant information about runtime activity.

### Diagnosing WLS-VE Issues

Issues with WLS-VE not specifically associated with WLS or with LiquidVM, can probably be traced to configuration errors. This section will help you identify the problem and figure out what caused it and how to resolve it. If you cannot find the solution here, collect the necessary information about your system, as described in Reporting a Problem to BEA Support, and open a case with BEA Support.

Note: | If you have configured your installation using an NFS share, you may encounter certain error conditions related to the NFS configuration. Common NFS error conditions are described in the WLS-VE version 1.0 Installation and Configuration Guide.

The most common error conditions you might encounter are:

- “Could not find the disk” Error
- Server Shuts Down Soon After Startup
- Server Shuts Down Soon After Startup
- “netSend failed: -3” Error
- “Configured IP [...] in use by MAC” Error

#### “Could not find the disk” Error

Symptom: When you launch your instance and you get the following output in your OS console window:

    Starting WLS-MyServer. connect...configure...create...
    Could not find the disk: [storage2] wlsve/isoName.iso

Problem: When the WLS-VE instance was created, VMware could not find the ISO image needed to boot up WLS-VE.

Solution:

Check that you have uploaded the WLS-VE ISO image to the ESX server.

Verify that the `bea.lvm.info` file in your home directory points to the correct location on the ESX server. You can do this either by manually editing the `bea.lvm.info` file in your favorite editor or by rerunning the LiquidVM Configuration Wizard (`server1001ve12`\tools\`lvm_configwizard.cmd` or `.sh`), as described in Configuring LiquidVM Connection Parameters

Confirm that the ISO image exists, using the LiquidVM Configuration Wizard:

1. Select VM Host.
2. Select the Configuration tab and note the Datastore Name.
3. Select Browse Datastore and confirm that `wlsve1001.iso` is available in your datastore.

#### Server Shuts Down Soon After Startup

Symptom: The server shuts down soon after startup and a LiquidVM log file named `WLS-``<servername>``.lvm.out` appears in the domain directory on the local disk. In that file, you find the following:

    <Jan 9, 2008 6:50:23 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic
    Server 10.0 MP1 Fri Dec 7 01:21:28 EST 2007 1023546 >
    <Jan 9, 2008 6:50:30 PM EST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
    Enter username to boot WebLogic server:The application tried to read from keyboard (stdin). However,
    reading from keyboard is not possible when running LiquidVM.
    Please be aware that this could result in unexpected behaviour
    if the application really depends on keyboard input working
    <Jan 9, 2008 6:50:30 PM EST> <Error> <Security> <BEA-090783> <Server is
    Running in Development Mode and Native Library(terminalio) to read the
    password securely from commandline is not found.>
    <Jan 9, 2008 6:50:30 PM EST> <Notice> <WebLogicServer> <BEA-000388> <JVM
    called WLS shutdown hook. The server will force shutdown now>
    <Jan 9, 2008 6:50:30 PM EST> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>
    <Jan 9, 2008 6:50:30 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Problem: You did not provide a user name and password either in the `security` directory (no `boot.properties` file) of the Administration Server's root directory or in the start script using the `WLS_USER` and `WLS_PW` properties. WLS-VE does not support normal keyboard input, so you cannot enter a user name and password on the keyboard.

Solution: Add a `boot.properties` file to the `security` directory of the Administration Server's root directory (see Creating a Boot Identity File for an Administration Server in Managing Server Startup and Shutdown) or add a user name and password in the start script and relaunch your WLS-VE server.

#### “netSend failed: -3” Error

Symptom: You receive the following error message on the Virtual Center console:

    000000 [rpcconn    WRN] netSend failed: -3
    000000 [rpcconn    WRN] Rpc call failed
    000000 [rpc        WRN] Rpc request failed: 3
    000000 [rpc        WRN] rpcDoReqeust returned 3
    000000 [rpc        WRN] rpcCall 3 returned 8549398

Problem: Your network configuration is incorrect.

Solution: In the start-up script, check your static IP address, your gateway, and your netmask and verify that they are correct. If they are not, obtain the correct information and enter it in the respective property.

#### “Configured IP [...] in use by MAC” Error

Symptom: When you attempt to start a server, you receive this message:

    000000 [net   WRN] Configured IP [172.18.134.55] in use by MAC: 00:50:56:a0:
    06:96
    000001 [net   WRN] Network stack initialization FAILED: 98

Problem: Someone is already using the IP address you have specified.

Solution: Another running VM might be using the same IP address. Do the following:

- Verify that you typed the IP address correctly.
- Verify that none of your running VMs already use that IP address.

If neither of the above is the case, someone is using your IP address. Because finding out who that person is might be difficult, contact your system administrator to obtain another IP address.

### Diagnosing WLS Issues

WLS-VE can encounter the same type of server-related issues that can occur when running non-virtualized WLS. This section provides an overview of the kinds of WLS issues you should watch for when running WLS-VE.

#### Performance Issues

Often, a problem with WLS is the result of poor tuning. For example, pool sizes (such as pools for JDBC connections, Stateless Session EJBs, and MDBs) that do not maximize concurrency for the expected thread utilization can adversely affect performance. Similarly, applications that handle large amounts of data per request will experience a boost in performance if the chunk size—that is, a unit of memory that the WLS network layer uses to read data from and write data to sockets—on both the client and server sides can be increased, a process called tuning the chunk size.

You can find tuning and performance guidelines in WebLogic Server Performance and Tuning.

Specific tuning guidelines for WLS-VE are provided in Tuning the WLS-VE LiquidVM Kernel.

#### Server Failure

A server instance can fail and different events can lead to this failure. Often one failure condition leads to another. Loss of power, hardware malfunction, OS crashes, network partitions, and unexpected application behavior can all contribute to the failure of a server instance. Even in a clustered environment, server instances may fail periodically and you must be prepared for the recovery process. See Avoiding and Recovering From Server Failure in Managing Server Startup and Shutdown for information on dealing with server failure.

#### Clustering Issues

A number of cluster issues can affect the performance of WLS. These issues can occur for many reasons, including versioning errors, multicast addressing issues, errors or misspellings in start-up commands, and even a poorly-tuned memory management systems. You can find guidelines for troubleshooting cluster issues in Troubleshooting Common Problems in Using WebLogic Server Clusters.

#### Other WLS Issues

Other, non-specific issues can also occur with WLS. When these issues occur, they usually generate an error message with an associated error code. The Index of Messages by Message Range provides descriptions, possible causes, and corrective actions for all WLS error conditions.

### Diagnosing LiquidVM Issues

Issues that do not originate with WLS may occur in LiquidVM and are typical to the kinds of issues you might encounter in non-virtualized JVMs. Issues such as these are documented in the BEA JRockit Diagnostics Guide (BEA JRockit is the JVM component of LiquidVM). This topic provides information for either resolving the problem yourself or mining the necessary information required to open a case with BEA Support.

The types of LiquidVM issues you might encounter when running WLS-VE are:

- System crashes occur when the entire system shuts down involuntarily and usually without warning. See The System is Crashing.
- System freezes occur when the application stops answering requests but the process is still there. See BEA JRockit is Freezing.
- Slow startups usually occur when BEA JRockit’s optimizing compiler must run extensively to ensure that the most efficient code possible is compiled. See BEA JRockit Starts Slowly.
- Poor performance usually occurs when your application experiences poor throughput. This usually indicates that the memory management system has not been tuned for optimal performance. See Low Overall Throughput.
- Occasional slow response times usually indicate that transactions are taking too long to execute, a bottleneck most often caused by garbage collection pause times lasting too long. See Long Latencies.
- Performance degrading after the application has been running is characterized by your application’s behavior: it may be working fine early in its run, but after a while it may report the wrong results, or throw exceptions where it shouldn’t, or it simply crashes or hangs at roughly the same time each time you run it. See BEA JRockit’s Performance Degrades Over Time.

Note that in most UNIX OSes there is a file descriptor limit that limits the number of files and sockets you can have open. LiquidVM does not have such limits so there is no need (and no way) to set a file descriptor limit.

For complete information on BEA JRockit problem determination and resolution, see BEA JRockit Tools in the BEA JRockit Diagnostics Guide.

## Handling Suspend Files

When WLS-VE crashes, the VM goes into a state of suspension. A pause button will appear on the VirtualCenter and information about the crash will be written to the console. When a suspend file is created, do the following:

1. `tar-gzip` the suspend file. You will find it on the VM’s home directory on the ESX server; it will have a filetype of `.vmss`.
2. Copy the `tgz` file from the ESX server to your normal environment (for example, your `My Documents/` folder).
3. Upload the `tgz` file to BEA Support.

Be aware that you might not realize that your machine has actually crashed when it suspends. You should avoid resuming execution, because you might lose critical information that would be helpful in diagnosing the issues causing the crash. You should also be aware that suspend files are large and so it might not be efficient to copy from the ESX server.

## Displaying Version Information

A critical piece of information that Support will need to help diagnose any issues you report to them is the version number. You can find this number in the file `WLSVE_VERSION`, which is located in the `server1001ve12/` directory.

Open this file to find the version number; for example:

10.0.0.1ve1.2-b16

## Reporting a Problem to BEA Support

If your machine crashing, running slowly, or returning unpredictable results you may have a problem with WLS-VE. Being able to identify what kind of problem you are experiencing will help you know what kind of information you need to include when you open the trouble report.

If you determine that you need to open a case with BEA support, this section discusses what you need to do before opening the case to ensure that you supply the support personnel assigned to your issue as complete picture of what is wrong as possible. The more information you can provide, the more quickly will the support staff be able to resolve your issue.

### Verify That You Are Running a Supported Configuration

Before submitting a bug, verify that the environment in which the problem was found is a supported configuration. See the Supported Configurations page for WLS-VE.

### Collect Enough Information to Define Your Issue

In addition to testing with the latest update release, use the following guidelines to prepare for submitting a trouble report:

1. Collect as much relevant data as possible. For example, generate a thread-dump in the case of a deadlock, or locate the core file (where applicable) and `hs_err` file in the case of a crash. In all cases it is important to document the environment and the actions performed just before the problem is encountered.
2. Where applicable, try to restore the original state and reproduce the problem using the guidelines provided in Troubleshooting WLS-VE. This helps to determine if the problem is reproducible or an intermittent issue.
3. If the issue is reproducible, try to narrow the problem. In some cases, a bug can be demonstrated with a small standalone test case. Bugs demonstrated by small test cases will typically be easy to diagnose when compared to test cases that consists of a large complex application.
4. Search the bug database to see if the bug, or similar bugs, have been reported. If the bug has already been reported, the bug report may have further information. For example, if the bug has already been fixed, it will indicate the release that the bug was fixed in. The bug may also contain information, such as a workaround or include comments in the evaluation that explain, in further detail, the circumstances that cause the bug to arise.

If you conclude that the bug has not already been reported, then it is important to submit a new bug.