C H A P T E R 3

Troubleshooting the File System Manager Software

This chapter describes how to troubleshoot issues that might arise with the File System Manager software.

This chapter contains the following sections:

- Log and Trace Files

- File System Manager Messages

- Remote Procedure Call (RPC) Daemon Information

##

Log and Trace Files

The File System Manager software automatically enables logging when it is installed, but you must enable tracing manually. To enable tracing for File System Manager, follow the instructions in Tracing.

Log rotation is not supported for log or trace files.

TABLE 3-1 lists the files that the File System Manager uses for logging and tracing.

TABLE 3-1 File System Manager Log and Trace Files
Activity | File Location | Created by the User?
--- | --- | ---
File System Manager logging | `/var/tmp/fsmgr.overall.log` | Yes
File System Manager logging | `/var/log/webconsole/fsmgr.log` | No
Tomcat web console logging | `/var/log/webconsole/console_debug_log` | No
Tracing for File System Manager and native code | `/var/log/webconsole/fsmgr.trace_syslog` | Yes

The following sections describe the log and trace files.

### File System Manager Logging

To generate an overall File System Manager troubleshooting report, run the `/opt/SUNWfsmgr/bin/fsmgr_report` command to create the `/var/tmp/fsmgr.overall.log` file.

This log file contains general system information such as OS version, host name and environment variables. It also contains package and version information for the software packages that support File System Manager, such as Java and Tomcat. It includes configuration files that impact or are modified by File System Manager.

The log file also contains data from the following File System Manager log files:

- `/var/log/webconsole/console_debug_log`

- `/var/log/webconsole/localhost_log`

- `/var/log/webconsole/fsmgr.log`

- `/var/log/webconsole/fsmgr.trace_syslog`

The File System Manager software creates the `fsmgr.log` log file when the application starts. It records information about operations that the user performs, and whether those operations were successful. Do not delete or modify this file; doing so will cause logging to stop. When the web server restarts, it erases the contents of this file and creates a new `fsmgr.log` file.

The File System Manager software uses an additional file, `/var/webconsole/fsmgr.log.lck`, to ensure that only one process at a time writes to the log file. Do not delete or modify this lock file.

### Web Server Logging

The Sun Common Console Framework creates the `/var/webconsole/console_debug_log` file. It includes console-specific information such as environment variable settings that the console uses and a record of users logged in to the console.

If this file becomes too large, you can delete this file. The system creates another instance of this file the next time the web server restarts.

### File System Manager Portal Agent Configuration and Log Files

When the File System Manager software is installed, the File System Manager Portal agent is also installed. This application acts as an information source for the Sun StorEdge Management Portal application. By default, the File System Manager Portal agent is disabled. You should enable it only if you are using the Sun StorEdge Management Portal software.

The agent uses the same underlying software as File System Manager. The following files are used for configuring and logging data from the File System Manager Portal agent:

- `/etc/opt/SUNWfsmgr/agent/conf.sh` - The configuration script that is used for starting the Tomcat process. It defines location of Tomcat, Java, and other critical components.

- `/var/opt/SUNWfsmgr/agent/tomcat/logs` - A directory containing the following log files:

- `catalina.out` - The general log file. It contains the log message output from both Tomcat and the agent servlet. Any errors cause log messages to be written to this file.

- `fsmgr.`date-stamp`.log` - The application and servlet log file. It contains messages specific to the loading and the running of the agent servlet. It also contains stack trace and fatal error information from the underlying software.

To verify that the File System Manager Portal agent is running, check the `catalina.out` log file or use the `ps` and `grep` commands to find the agent process. For example:

    # /usr/ucb/ps -augxww | grep SUNWfsmgr/agent/tomcat

### Tracing

The File System Manager trace file records the following information:

- Messages regarding whether operations were successful.

- Functions invoked with the application stack. These can be verbose.

- Messages that are important to developers for debugging purposes.

Tracing is not enabled by default.

To Enable Tracing for File System Manager and Native Code

The `syslog` daemon performs detailed tracing for File System Manager and for native code. Use the following procedure to enable detailed tracing.

1. Use the `touch`(1) command to create the trace file.

For example:

    # touch /var/log/webconsole/fsmgr.trace_syslog

2. Use `vi`(1) or another editor to add the following line to the `/etc/syslog.conf` file:

    local6.debug   /var/log/webconsole/fsmgr.trace_syslog

Use a tab character to separate the two fields in this line.

3. Type the following command:

    # pkill -HUP syslogd

4. (Optional) Enable trace file rotation.

Trace files can become very large. Use `logadm`(1M) to manage the trace file for File System Manager.

Note - You cannot use the `log_rotate.sh`(1M) script to manage the File System Manager trace file.

To Enable Tracing or to Adjust the Tracing Level

Use the following command to enable tracing or to adjust the tracing level:

    # /opt/SUNWfsmgr/bin/fsmgr trace trace-level

For trace-level, specify one of the values shown in TABLE 3-2.

TABLE 3-2 Arguments for trace-level
trace-level Argument | Tracing Requested
--- | ---
`off` | Disables tracing.
`1` | Enables tracing for very important messages only. This includes severe errors that occur within the application.
`2` | Enables tracing for moderately important messages. This includes level 1 messages as well as debugging statements within the application that are useful for developers.
`3` | Enables tracing for all messages. This includes level 1 and level 2 messages as well as entry and exit points of functions within the application on the stack.

You can enable and disable tracing dynamically during run time by using the `fsmgr`(1M) command.

#### File System Manager Portal Agent Tracing

The File System Manager Portal agent acts as an information source for the Sun StorEdge Management Portal application. The agent uses the same tracing mechanisms as File System Manager, and both applications write to the same trace output file. When you enable tracing, it is enabled for both File System Manager and the agent. If both the agent and File System Manager are running simultaneously, it can be hard to decipher the tracing output. If you need to enable tracing, it is therefore best to have either File System Manager or the agent running, but not both.

If you enable tracing and do not see any tracing output for the agent, check the access permissions on the trace log file. The agent application runs as `root`, so you need to verify that `root` has access to write to the trace log file.

##

File System Manager Messages

This section shows some of the messages you might see when using the File System Manager software.

- An unrecoverable error occurred during the page display. If the problem persists, please restart the web server.

Click the HOME button to return to the Server Selection page, which is the default page of the File System Manager application.

If the system cannot display the Server Selection page, go to the web server and enter the following command to restart the it:

    # /usr/sbin/smcwebserver restart

Contact your Sun Support representative if the problem persists.

- HTTP 500 Internal server error

Go to the web server and run the following command to restart it:

    # /usr/sbin/smcwebserver restart

Contact your Sun Support representative if the problem persists.

- The page cannot be displayed.

Go to the web server and run the following command to restart it:

    # /usr/sbin/smcwebserver restart

Contact your Sun Support representative if the problem persists.

- Starting Sun(TM) Web Console Version 2.2.5.
  Startup failed. See /var/log/webconsole/console_debug_log for detailed error information.

Examine the contents of the following file on the web server:

`/var/log/webconsole/console_debug_log`

If the log says the port (`6789`) is in use by some other process, issue the following commands:

    # pkill -9 noaccess
    # /usr/sbin/smcwebserver restart

Contact your Sun Support representative if the problem persists.

- Failed to create the filesystem
  mount_samfs: fopen(mnttab) error: : Too many open files

The system generates this message if you are trying to create a file system with a large number of LUNs. To remedy this problem, follow these steps:

a. On the file system server, use the `ps`(1) and `grep`(1) commands to find the process ID for the `fsmgmtd` process.

For example:

    # ps -ef | grep fsmgmtd

b. Use the `plimit`(1) command to increase the descriptors for the process.

For example:

    # plimit -n 512 process-id

For process-id, specify the process number.

c. Create the file system.

##

Remote Procedure Call (RPC) Daemon Information

The following procedure can help you obtain troubleshooting information for the remote procedure call (RPC) daemon, `fsmgmtd`(1M).

### To Determine Whether the RPC Daemon Is Running

1. Log in to the Sun StorEdge SAM-FS server.

2. Become superuser.

3. Display status information for the File System Manager daemon (`fsmgmtd`):

Issue the following command to display:

    # /opt/SUNWsamfs/sbin/fsmadm status

If the daemon is not running, it does not display its status. Enter the following command to start the daemon:

    # /opt/SUNWsamfs/sbin/fsmadm config -a

This command also enables the daemon to restart automatically if it dies.

Copyright © 2006, Sun Microsystems, Inc. All Rights Reserved.