C H A P T E R 4

Troubleshooting the Server and Restoring ILOM Defaults

This chapter introduces the diagnostic tools you can use to troubleshoot or monitor the performance of your server. It also includes information about how to restore the service processor (SP) password and serial connection defaults in ILOM, as well as how to restore your ILOM SP firmware.

- Troubleshooting the Server

- Diagnostic Tools

- Diagnostic Tool Documentation

- Restoring ILOM Defaults

- Accessing the Preboot Menu

- Restoring the Factory Default ILOM Root Password

- Restoring ILOM Access to the Serial Console

- Restoring the SP Firmware Image

- Preboot Menu Command Summary

##

4.1 Troubleshooting the Server

The server module and its accompanying software and firmware contain many diagnostic tools and features that can help you complete the following tasks:

- Isolate problems when a failure occurs on a customer-replaceable unit (CRU)

- Monitor the status of a functioning system

- Exercise the system to disclose more subtle problems that might be hardware related

A suite of diagnostic tools, each of which has its own specific strengths and applications is available for troubleshooting server issues. For more information about which tool might be best to use for your purpose, and where to locate information about these tools, see:

- Diagnostic Tools

- Diagnostic Tool Documentation

### 4.1.1 Diagnostic Tools

There is a wide selection of diagnostic tools for use with your server. These tools range from the SunVTS software, a comprehensive validation test suite, to log files in ILOM that might contain clues helpful in identifying the possible sources of a problem, and the fault management function in ILOM that enables you to identify a faulted component as soon as the fault occurs.

The diagnostic tools also include standalone software packages, firmware-based tests like power-on self-test (POST), U-Boot tests, or Pc-Check tests, and hardware LEDs that tell you when the system components are operating.

TABLE 4-1 summarizes the variety of diagnostic tools that you can use when troubleshooting or monitoring your server.

TABLE 4-1 Summary of Server Diagnostic Tools
Diagnostic Tool | Type | What It Does | Accessibility | Remote Capability
--- | --- | --- | --- | ---
Integrated Lights Out Manager (ILOM) | SP firmware | Monitors environmental conditions, generates alerts, performs fault isolation, and provides remote console access. | Can function on standby power and when the operating system is not running. | Designed for remote and local access.
Preboot Menu | SP firmware | Enables you to restore some of ILOM defaults (including firmware) when ILOM is not accessible. | Can function on standby power and when operating system is not running. | Local, but remote serial access is possible if the SP serial port is connected to a network-accessible terminal server.
LEDs | Hardware and SP firmware | Indicate status of overall system and particular components. | Available when system power is available. Note - Fault Remind LEDs are available for a few minutes after the power is removed. | Local, but sensor and indicators are accessible from ILOM web interface or command-line interface (CLI).
POST | Host firmware | Tests core components of system: CPUs, memory, and motherboard I/O bridge integrated circuits. | Runs on startup. Available when the operating system is not running. | Local, but can be accessed through ILOM Remote Console.
U-Boot | SP firmware | Initializes and tests aspects of the SP prior to booting the ILOM SP operating system. Tests SP memory, SP, network devices and I/O devices. | Can function on standby power and when operating system is not running. | Local, but remote serial access is possible if the SP serial port is connected to a network-accessible terminal server.
Pc-Check | SP firmware (or on Tools and Driver DVD for systems without SP) | DOS-based utility that tests all motherboard components (CPU, memory, and I/O), ports, and slots. | Runs on startup. Available when the operating system is not running. | Remote access through ILOM Remote Console for servers shipped with SP. or Local access for servers shipped without an SP (via Tools and Driver DVD).
Solaris commands | Operating system software | Displays various kinds of system information. | Requires operating system. | Local, and over network.
SunVTS | Diagnostic tool standalone software | Exercises and stresses the system, running tests in parallel. | Requires operating system. Install SunVTS software separately. | View and control over network.

### 4.1.2 Diagnostic Tool Documentation

TABLE 4-2 identifies where you can find more information about Sun diagnostic tools.

TABLE 4-2 Summary of Documentation for Sun Diagnostic Tools
Diagnostic Tool | Where to Find Information
--- | ---
ILOM | Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943 Locate the latest version of these guides at these URLs: `http://docs.sun.com/app/docs/prod/nt6270.m2.srvrmod?l=en#hic` `http://docs.sun.com/apps/docs/prod/int.lights.mgr30#hic`
Preboot Menu | Restoring ILOM Defaults
LEDs; or System indicators and sensors | This service manual Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943 Locate the latest version of these guides at: `http://docs.sun.com/app/docs/prod/nt6270.m2.srvrmod?l=en#hic`
POST | About the BIOS Default BIOS Power-On Self-Test (POST) Events BIOS POST F1 and F2 Errors How BIOS POST Memory Testing Works
U-Boot or Pc-Check | Oracle x86 Servers Diagnostics Guide, 820-6750 Locate the latest verson of this guide at: `http://docs.sun.com/app/docs/prod/nt6270.m2.srvrmod?l=en#hic`
Solaris commands | Locate the latest Solaris command information for Solaris 10 at: `http://docs.sun.com/app/docs/doc/817-0550/eqxqt?a=view`
SunVTS | Locate the latest documentation for SunVTS at: `http://docs.sun.com/app/docs/prod/test.validate`

##

4.2 Restoring ILOM Defaults

You can restore the factory defaults for the following ILOM features:

- ILOM root password

- Serial port connection

To restore these factory defaults, you must use the Preboot Menu utility that is shipped installed on your server. The Preboot Menu enables you to address changes to some of ILOM’s settings while ILOM is not currently running. In addition to restoring factory defaults for the root password and serial connection settings, the Preboot Menu enables you to restore the SP firmware image on your system.

For more information about how to use the Preboot Menu to restore settings in ILOM, see these sections:

- Accessing the Preboot Menu

- Restoring the Factory Default ILOM Root Password

- Restoring ILOM Access to the Serial Console

- Restoring the SP Firmware Image

- Preboot Menu Command Summary

### 4.2.1 Accessing the Preboot Menu

To access the Preboot Menu, you must reset the SP and interrupt the boot process. You can interrupt the ILOM boot process by either:

- Using the Locate button on the local server.

For details about the requirements for this local access method, see Prerequisites for Accessing the Preboot Menu.

Or

- Typing `xyzzy` during a pause in the bootstrap process from a remote serial terminal.

For details about the requirements for this remote serial access method, see Prerequisites for Accessing the Preboot Menu.

Some Preboot Menu settings must be configured first, and until they are, you must use the Locate button method to access the Preboot Menu.

For detailed instructions for accessing the Preboot Menu from a local or remote connection, see the following sections:

- Prerequisites for Accessing the Preboot Menu

- Access to the Preboot Menu

- Edit Preboot Menu for Remote Serial Access

#### 4.2.1.1 Prerequisites for Accessing the Preboot Menu

Ensure that the applicable requirements are met prior to accessing the Preboot Menu from either a local or remote connection.

- For local access to the Preboot Menu (using Locate button)

You must connect a terminal or console capable of running terminal emulation software to the serial port on the server using a dongle cable.

For more information about how to attach local devices to the server using a dongle cable, see Attaching Devices to the Server Module.

For instructions for accessing the Preboot Menu by using the Locate button, see Access to the Preboot Menu.

- For remote serial access to the Preboot Menu (typing `xyzzy` during bootstrap)

Prior to accessing the Preboot Menu remotely, you must set the `bootdelay` and `check_physical_presence` settings in the Preboot Menu to enable remote serial access. To configure these settings for the first time, you need to:

a. Access the Preboot Menu using the Locate button on the local server as described in Access to the Preboot Menu.

b. Edit the settings in the Preboot Menu to enable remote access as described in Edit Preboot Menu for Remote Serial Access.

c. Use a remote serial terminal or console running terminal emulation software to access the Preboot Menu remotely.

Note - You cannot use an SSH, or a remote KVMS session to access the Preboot Menu remotely.

### Access to the Preboot Menu

1. Ensure that the requirements in Prerequisites for Accessing the Preboot Menu are met.

2. Reset ILOM.

For example:

- From the server SP ILOM, enter the command:

`->` `reset /SP`

- From the CMM ILOM, enter the command:

`->` `reset /CH/BL` n`/SP`

Where n is the slot number of the server module.

- Temporarily remove power from the server module by partially removing it from its slot, then reseating it.

ILOM reboots and messages begin scrolling on the screen.

3. To interrupt the ILOM boot process, perform one of the following actions when the SP is resetting:

- Press and hold the Locate button on the server module front panel, until the Preboot Menu appears.

Or

- Type in `xyzzy` when you see the message:

`Booting linux in` n `seconds...`

Note - You cannot interrupt the ILOM boot process by typing `xyzzy` until you have configured the settings as described in Edit Preboot Menu for Remote Serial Access.

The ILOM Preboot Menu appears.

    Booting linux in 10 seconds...

                            ILOM Pre-boot Menu
                            ------------------
    Type "h" and [Enter] for a list of commands, or "?" [Enter] for
    command-line key bindings.  Type "h cmd" for summary of 'cmd' command.

    Warning: SP will warm-reset after 300 seconds of idle time.
      Set 'bootretry' to -1 to disable the time-out.

    Preboot>

4. You can perform any of the following tasks or type `boot` to exit the Preboot Menu.

- Enable remote access to the Preboot Menu.

For instructions, see Edit Preboot Menu for Remote Serial Access.

- Restore the factory default to the ILOM root password.

For instructions, see Restore ILOM Root Password to Factory Default Using the Preboot Menu.

- Restore the serial connection between ILOM and a host console.

For instructions, see Restore Access to the Serial Console Using the Preboot Menu.

- Restore the SP firmware image.

For instructions, see Restore the SP Firmware Image Using the Preboot Menu.

- Review the commands supported in the Preboot Menu.

For command details, see Preboot Menu Command Summary.

### Edit Preboot Menu for Remote Serial Access

1. Access the Preboot Menu as described in Access to the Preboot Menu.

2. At the Preboot> prompt, type `edit`.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

- To change a setting, type the new value, then press Enter.

- To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the `bootdelay` setting appears.

4. To change the `bootdelay` setting, type `3`, `10`, or `30`, then press Enter.

This value (`3`, `10`, or `30`) specifies the number of seconds the SP boot process delays while waiting for your input.

The Preboot Menu redisplays the `bootdelay` setting with the new value.

5. Press Enter to return to Preboot Menu selections.

The Preboot Menu selections appear.

6. Press Enter to move through the settings until the `check_physical_presence` setting appears.

To change the `check_physical_presence` setting, type `no`, then press Enter.

The Preboot Menu displays the `check_physical_presence` setting with the new value.

7. Press Enter for the new value to take effect.

The Preboot Menu asks you to confirm your changes.

`Enter ‘y[es]’` `to commit changes:` `[no]`

8. Type `y` to save your changes and exit the edit session.

If you want to exit the edit session without saving your changes, type `n`.

The following example shows an edit session where the `bootdelay` and `check_physical_presence` settings are changed.

Note - For a list of other settings you can edit in the Preboot Menu, see Edit Mode Settings in Preboot Menu.

    Preboot> edit

    Press Enter by itself to reach the next question.
      Press control-C to discard changes and quit.

     Values for baudrate are {[ 9600 ]| 19200 | 38400 | 57600 | 115200 }.
      Set baudrate?                [9600]
     Values for serial_is_host are {[ 0 ]| 1 }.
      Set serial_is_host?          [0]
     Values for bootdelay are { -1 | 3 | 10 | 30 }.
      Set bootdelay?               [30] 10
      Set bootdelay?               [10]
     Values for bootretry are { -1 | 30 | 300 | 3000 }.
      Set bootretry?               [<not set>]
     Values for preferred are {[ 0 ]| 1 }.
      Set preferred?               [<not set>]
     Values for preserve_conf are {[ yes ]| no }.
      Set preserve_conf?           [yes]
     Values for preserve_users are {[ yes ]| no }.
      Set preserve_users?          [no]
     Values for preserve_password are {[ yes ]| no }.
      Set preserve_password?       [yes]
     Values for check_physical_presence are {[ yes ]| no }.
      Set check_physical_presence? [no] no
      Set check_physical_presence? [no]
     Enter 'y[es]' to commit changes: [no] y
    Summary: Changed 2 settings.
    Preboot>

#### 4.2.1.2 Edit Mode Settings in Preboot Menu

In addition to changing the settings required in the Preboot Menu to enable remote access, you can also change other edit mode settings in the Preboot Menu. For a list of these settings, see TABLE 4-3.

TABLE 4-3 Edit Mode Preboot Menu Command Settings
Setting | Description
--- | ---
`baudrate` | Sets the baud rate of the serial port. Selections include 9600,19200, 38400, 57600, and 115200.
`serial_is_host` | If this is set to 0, the serial port connects to the ILOM. If this is set to 1, the serial port connects to the host. For more details, see Restoring ILOM Access to the Serial Console.
`bootdelay` | The number of seconds the bootstrap process waits for the user to enter `xyzzy` before booting the SP. Selections include `3`, `10`, or `30.`
`bootretry` | The number of seconds the Preboot Menu waits for user input before timing out and starting the SP. Set to -1 to disable the time-out.
`preferred` | Unused
`preserve_conf` | Setting this to `no` duplicates the function of the `unconfig ilom_conf` command, which resets many ILOM configuration settings, but preserves SP network, `baudrate`, and `check_physical_presence` the next time the SP is booted.
`preserve_users` | Setting this to `no` duplicates the function of the `unconfig users` command, which resets user information to the default values the next time the SP is booted.
`preserve_ password` | Setting this to `no` duplicates the function of the `unconfig password` command, which resets the root password to the default value the next time the SP is booted.
`check_physical_presence` | If this is set to `Yes`, you must press and hold the Locate button to interrupt the SP boot process. If it is set to `No`, the boot process prompts you to interrupt it. See Edit Preboot Menu for Remote Serial Access for details.

### 4.2.2 Restoring the Factory Default ILOM Root Password

The ILOM root password grants you access to the ILOM web interface or command-line interface (CLI) on the SP or chassis monitoring module (CMM). If you forget the root password, you can use the Preboot Menu to restore the password to the factory default, `changeme`.

### Restore ILOM Root Password to Factory Default Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. In Preboot Menu, type:

`Preboot>` `unconfig password`

`Setting ‘preserve_password’ to ‘no’ for the next boot of ILOM.`

3. Reset the SP by typing:

`Preboot>` `boot`

The Preboot Menu exits and the SP restarts.

After restarting the ILOM SP, the value for the root password (on the ILOM SP and CMM) is set to `changeme` when the SP is finished booting.

### 4.2.3 Restoring ILOM Access to the Serial Console

In the event that the serial connection between ILOM and a host becomes unavailable, you can restore access to the serial port connection by reconfiguring the host as the external serial port owner in either the ILOM web interface or CLI, or in the Preboot Menu.

To determine which interface to use when restoring the serial connection between ILOM and a host console, consider the following:

- If a network connection is available, you should use the ILOM web interface or CLI to restore ILOM’s access to the serial console.

For instructions, see the procedure for “Switching Serial Port Output Between SP and Host Console” in the Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Netra X6270 M2 Server Module, 821-0943.

- If a network connection is unavailable, you should use the procedure in Restore Access to the Serial Console Using the Preboot Menu to restore ILOM access to the serial console.

### Restore Access to the Serial Console Using the Preboot Menu

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type `edit`.

The Preboot Menu enters edit mode.

In edit mode, the Preboot Menu displays its selections one-by-one, offering you a chance to change each one.

- To change a setting, type the new value, then press Enter.

- To skip to the next setting, press Enter.

3. Press Enter to move through the settings until the `serial_is_host` setting appears.

To change the `serial_is_host` setting, type `0`, and then press Enter.

The Preboot Menu appears with the new value for the `serial_is_host` setting.

4. Press Enter to display the Preboot Menu selections.

The Preboot Menu settings appear.

5. Press Enter to scroll through the settings until the Preboot Menu asks you to confirm your changes.

`Enter ‘y[es]’ to commit changes: [no]`

6. Type `y` to confirm your change and exit the edit session.

The Preboot Menu displays this message

`Summary: Changed 1 settings`

`Preboot>`

7. To exit the Preboot Menu, type `boot`.

### 4.2.4 Restoring the SP Firmware Image

If ILOM is available, you should always use the ILOM web interface or CLI to restore the firmware image. For instructions about how to restore the firmware image using either the ILOM web interface or CLI, see the Oracle Integrated Lights Out Manager (ILOM) 3.0 Documentation Collection. If ILOM is unavailable, you can use the Preboot Menu to restore the SP firmware image.

Note - To restore the SP firmware image using the Preboot Menu, you must be an authorized Service technician to perform the procedure.

To use the Preboot Menu to restore the SP firmware image on the server, see the following sections:

- Prerequisites for Restoring SP Firmware Using the Preboot Menu

- Enabling Support in Preboot Menu to Recover the SP

- Restore the SP Firmware Image Using the Preboot Menu

#### 4.2.4.1 Prerequisites for Restoring SP Firmware Using the Preboot Menu

The following requirements must be met prior to restoring the SP firmware on your server using the Preboot Menu.

- You must be an authorized Service technician to perform this procedure.

- You must have a valid `.flash` firmware image file on a TFTP server.

Note - Restoring the SP firmware using the Preboot Menu requires a `.flash` file instead of a`.pkg` file that is typically used to update the SP using the ILOM interfaces.

Enabling Support in Preboot Menu to Recover the SP

You must enable support in the Preboot Menu to recover the SP firmware image prior to performing the steps in Restore the SP Firmware Image Using the Preboot Menu.

To enable support in the Preboot Menu to recover SP firmware, follow these steps.

1. Prepare the server for service by powering down the server, removing the server module from the system chassis, then removing the server cover.

For instructions, see the following sections:

- Powering On and Off the Server Module

- Remove Server Module From Chassis

- Removing or Installing Server Module Cover

2. Locate the J602 pins on the server module then place a jumper on pins 2 and 3. See FIGURE 4-1.

FIGURE 4-1 Placing Jumper on Pins

3. Replace the top cover, return the server module to the system chassis and power-on the server.

For instructions, see the following sections:

- Removing or Installing Server Module Cover

- “Installing the Server Module Into the Chassis” in the Sun Netra X6270 M2 Server Module Installation Guide, 821-0937.

- Powering On and Off the Server Module

4. Follow the instructions for restoring the SP firmware using the Preboot Menu in Restore the SP Firmware Image Using the Preboot Menu.

Note - The Preboot Menu firmware recovery process must be performed by an authorized Service technician and you must have a valid `.flash` file to perform the procedure.

5. After you restore the SP image using the Preboot menu, perform the following steps to remove the J602 jumper from the server and to return the server to normal operation.

a. Power off the server module.

See Power Off and Verify Server Is Ready for Removal From Chassis.

b. Remove the server module from the chassis.

See Remove Server Module From Chassis.

c. Remove the top cover from the server.

See Remove Cover From Server.

d. Remove the jumper from J602. See FIGURE 4-1 for the location of the J602 jumper.

e. Replace the top cover.

See Install Cover on Server.

f. Return the server module into the chassis.

See “Installing the Server Module Into System Chassis” in the Sun Netra X6270 M2 Server Module Installation Guide, 821-0937.

g. Power on the server.

See Power On the Server Module.

Restore the SP Firmware Image Using the Preboot Menu

To restore the SP firmware image using the Preboot Menu, follow these steps:

1. Access the Preboot Menu as described in Accessing the Preboot Menu.

2. At the Preboot> prompt, type:

| Caution - The use of the `net` `flash`command is reserved for use by authorized Service personnel only.

`net flash` IPaddress path/name`.flash`

Where:

- IPaddress is the IP address of a TFTP server.

- path is the path to the file relative to `/tftpboot.`

- name is the first part of the .`flash` file name.

For example:

`Preboot>` `net flash 10.8.173.25 images/system-rom.flash`

After a series of messages, the Preboot Menu prompt appears.

`Preboot>`

3. At the `Preboot>` prompt`,` type the `reset` command to restart the SP.

For example:

`Preboot>``reset`

The Preboot Menu exits and ILOM restarts.

### 4.2.5 Preboot Menu Command Summary

TABLE 4-4 identifies the Preboot Menu commands.

TABLE 4-4 Preboot Menu Commands
Command | Description
--- | ---
`boot` | Boots ILOM. The Preboot Menu exits and ILOM restarts. Note - This command executes a modified boot sequence that does not offer the choice to select the diagnostic level, or to interrupt the boot sequence and return to the Preboot Menu. To execute the normal boot sequence, use the `reset warm` command instead.
`vers` | Displays version information including the hardware type, board revision, ILOM revision, revisions of PBSW and recovery U-Boot. Shows the checksum integrity of the images, and the preference between redundant images.
help | Displays a list of commands and parameters.
`show` | Displays a list of SP settings.
`edit` | Starts an interactive dialog that prompts and changes settings one-by-one. See Edit Mode Settings in Preboot Menu for details.
`diag` | Runs the U-Boot diagnostic tests in manual mode. See the Oracle x86 Servers Diagnostics Guide, 820-6750 for more on U-Boot diagnostic tests.
`host` | Initiates various activities related to the host. `clearcmos` - Clears CMOS and BIOS passwords. `console` - Connects SP console to host serial console. Note - Type `Ctrl \ q` to quit. `show` - Shows information about the host state. `enable-on` - Enables the front-panel Power button, which is usually disabled unless ILOM is running. Caution - If you start the host when ILOM is off, the BIOS does not send error events, or power messages to the SP. This can cause all server modules to lose power. `hard-off` - Turns the host off.
`net` | `{` `config` `|` `dhcp |` `ping` `|` `flash` `}` `config` - Starts a dialog that enables you to change the ILOM network settings. `dhcp` - Changes the network addressing from static to DHCP. Note - You must set `ipdiscovery` `=` `dhcp` using the `net` `config` command first. `ping` - Sends a ping. `flash` - Downloads an ILOM firmware image. See Restoring the SP Firmware Image for details. Type `help net` command for more details on these commands.
`reset` | { `warm` `|` `cold` }. Resets the SP and the host. `warm` - Resets the SP without affecting a running host. `cold` - Resets the SP and the host. It has the effect of powering off the server module.
`unconfig` | `{ users` `|` `ilom_conf` `|` `most` `|` `all` `}` Causes ILOM to erase any configuration information and returns the values to defaults the next time ILOM boots. `users` - Resets all configured user information. `password` - Resets the ILOM root password to the default. See Restoring the Factory Default ILOM Root Password for more details. `ilom_conf` - Resets configuration settings but preserves SP network and `baudrate`, `preferred`, and `check_physical_presence`. `most` - Resets the SP data storage, but preserves SP network and `baud rate`, `preferred`, and `check_physical_presence` settings. `all` - Resets all SP data storage and settings. Booting ILOM restores other defaults. Note - None of these options erases the dynamic FRU PROMs.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.