Skip Headers

Oracle® Audit Vault Administrator's Guide 10g Release 2 (10.2.2) Part Number B25321-02 |
Home | Book List | Contents | Index | Contact Us

Previous | Next
| View PDF

# 6 Troubleshooting an Audit Vault System

This chapter provides troubleshooting information for administering an Audit Vault system. This chapter includes the following sections:

- Location of Audit Vault Server Log and Error Files

- Location of Audit Vault Agent Log and Error Files

- Troubleshooting Tips

## 6.1 Location of Audit Vault Server Log and Error Files

Table 6-1 shows the names and a description of the Audit Vault Server log and error files located in the Audit Vault Server `$ORACLE_HOME/av/log` directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur. Log files can be deleted at any time, except for the `avca.log` file, which can only be deleted when the Audit Vault Server is shut down.

Table 6-1 Name and Description of Audit Vault Server Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of errors encountered in agent initialization. This file can be deleted at any time.
`agent.out` | Contains a log of all primary agent-related operations and activity. This file can be deleted at any time.
`avca.log` | Contains a log of all AVCA commands that have been run and the results of running each command. This file can only be deleted after Audit Vault Server is shut down.
`av_client-``%g``.log.``n` | Contains a log of the agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 10 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. This file can be deleted at any time.
`avorcldb.log` | Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time.

Oracle Enterprise Manager stores its logs in the directory `Audit Vault_Server_Home``/``Host_Name``_``SID``/sysman/log`. The file `emdb.nohup` in this directory contains a log of activity for the Audit Vault Console, including graphical user interface (GUI) conversations, requests from the AVCTL utility and communication with the various Audit Vault agents. This information can be used to debug communication issues between the server and the agents.

## 6.2 Location of Audit Vault Agent Log and Error Files

Table 6-2 shows the names and a description of the Audit Vault Agent log and error files located in the Audit Vault Agent `$ORACLE_HOME/av/log` directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur.

Table 6-2 Name and Description of Audit Vault Agent Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of all errors encountered in agent initialization and operation. This file can be deleted at any time.
`agent.out` | Contains a log of all primary agent-related operations and activity. This file can be deleted at any time.
`avca.log` | Contains a log of all AVCA commands that have been run and the results of running each command. This file can be deleted at any time.
`avorcldb.log` | Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time.
`DBAUD_Collector_<source-name_prefix><source-id>.log` | Contains a log of collection operations for the DBAUD_Collector collector. This file can only be deleted after Audit Vault Agent is shut down.
`orcldb_osaud_<source name>.log` | Contains a log of all collection operations for the OSAUD_Collector collector. This file can only be deleted after Audit Vault Agent is shut down.
`av_client-``%g``.log.``n` | Contains a log of the agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 10 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. This file can be deleted at any time.
`sqlnet.log` | Contains a log of SQL*Net information.

The directory `Audit_Vault_Agent_Home``/oc4j/j2ee/home/log` contains the logs generated by the agent OC4J. In this directory, the file `AVAgent-access.log` contains a log of requests the agent receives from the Audit Vault Server. This information can be used to debug communication issues between the server and the agent.

Failed configuration commands are located in the Audit Vault Agent `$ORACLE_HOME/cfgtoollogs` directory, which includes the file, configToolFailedCommands. This file contains just the name of the failed command. See the `avca.log` or `avorcldb.log` file for additional information, including any associated errors and error messages.

## 6.3 Troubleshooting Tips

This section describes a number of troubleshooting scenarios that you might encounter with some of the Audit Vault components and how try to resolve each one. The scenarios are placed in the following groupings:

- Audit Vault Server

- Audit Vault Agent

- Audit Vault Collector

- Audit Vault Console

- Audit Vault in an Oracle Real Application Clusters (Oracle RAC) Environment

### 6.3.1 Audit Vault Server

This section describes Audit Vault Server problems that you might encounter.

Problem: Best way to tune Audit Vault Server performance when using the REDO collector.

Following an Audit Vault Server installation, the `streams_pool_size` initialization parameter is set to 150 MB. This parameter must be tuned to maximize REDO collector performance if you are going to be using this collector. In an Oracle Real Application Clusters (Oracle RAC) environment, this parameter must be tuned on all nodes because it is uncertain where the queue will be particularly after an instance startup.

Solution:

Typically, once a REDO collector is configured and started, let it run for a while. This will allow the autotuning feature of Oracle Database to allocate memory for the best database performance for the `streams_pool_size` parameter. Using AWR, check to see if STREAMS AQ has a flow control issue – enqueue being blocked. In the event that you notice that the performance, for example, is only 500 records being applied per second, it may become necessary to manually tune this parameter.

Assuming that you have at least 1 GB of physical memory in your Audit Vault Server system, set this parameter to 200 MB using the SQL command `ALTER SYSTEM SET STREAMS_POOL_SIZE=200;`. Monitor the performance again using AWR. You should achieve a record apply rate of 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB should be sufficient. If you using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster. Use the SQL command `ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=av` `n;`, where `n` is the number portion of the SID for each node in the cluster, for example, `av2`, `av3`, `av4`, and so forth, if that is your naming convention.

### 6.3.2 Audit Vault Agent

This section describes Audit Vault Agent problems that you might encounter.

Problem: While issuing an AVORCLDB setup command in the agent shell, you misenter the srcusr password in setting up the source on the agent. How do you recover from this problem?

In the agent shell, one of the last setup steps involves setting up the source with the agent using the AVORCLDB setup command. When entering the `-srcusr` argument, if you enter an incorrect password and invoke the command, an error message is returned indicating that the password is not recognized. Suddenly, you realize the source of the error as being a mistyped password.

Efforts to reenter the command using the correct password for the source user indicates that the credential already exists, so it cannot be entered again. How do you work around this problem so that you can use the setup command and the correct source user password?

Solution:

The incorrect credential that was added populates the avwallet in the Audit Vault Agent home. One workaround is to rename the avwallet file and create a new avwallet file. Next, you must add the agentuser credential. Finally, invoke the AVORCLDB setup command using the correct source user password. These steps follow:

1. In the Audit Vault Agent home, change directory to the `avwallet` directory.

       cd ../../network/admin/avwallet

2. Rename the `avwallet` file.

       mv avwallet/ avwallet.1

3. Create the `avwallet` file. The `-wpwd` argument can be omitted if the corresponding environment variable, `AVCA_WPWD` is set to `wallet password`. If the command-line argument `-wpwd` is specified, then the command-line argument overrides the environment variable. In this example, the environment variable is set and the `-wpwd` argument is omitted.

       avca create_wallet -wrl $ORACLE_HOME/network/admin/avwallet

4. Check to see that the `avwallet` file was created.

       ls -l avwallet

5. Check to see that no credentials exist that allow you to connect to the Audit Vault database.

       sqlplus /@av

   Note that the connection fails because the agent user credential does not exist.

6. Create the credential for the agent user. The `-wpwd` argument can be omitted if the corresponding environment variable, `AVCA_WPWD` is set to `wallet password`. If the command-line argument `-wpwd` is specified, then the command-line argument overrides the environment variable. The `-usr` argument can be omitted if the corresponding environment variable, `AVCA_USR` is set to `usr/password`. If the command-line argument `-usr` is specified, then the command-line argument overrides the environment variable. In this example, the environment variable is set and the `-wpwd` argument is omitted.

       avca create_credential -wrl $ORACLE_HOME/network/admin/avwallet -wpwd <passwd>
                              -dbalias <dbalias>

7. Check to see that the agent user credential exists, allowing you to connect to the Audit Vault database.

       sqlplus /@av

   The connection succeeds.

8. Invoke the AVORCLDB `setup` command in the Audit Vault Agent shell. The `-wpwd` argument can be omitted if the corresponding environment variable, `AVORCLDB_WPWD` is set to `wallet password`. The `-srcusr` argument can be omitted if the corresponding environment variable, `AVORCLDB_SRCUSR` is set to `source username/password`. If the command-line arguments `-wpwd` and -usr are specified, then the command-line arguments override the environment variable. In this example, these environment variables are set and the `-srcusr` and `-wpwd` arguments are omitted.

       avorcldb setup -srcname DBS1.US.ORACLE.COM

9. Check the `tnsnames.ora` file in the Audit Vault Agent home to see that it contains a SRCDBA1 alias.

       vi tnsnames.ora

       SRCDB1   = (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)
       (PORT=1521))
       (CONNECT_DATA=(SERVICE_NAME=src1.DBS1.US.ORACLE.COM)))

10. Check to see that the source user alias for the source database can connect to the Audit Vault database.

        sqlplus /@srcdba1

    Note that the connection succeeds. The `avwallet` file is current and working.

### 6.3.3 Audit Vault Collector

This section describes Audit Vault collector problems that you might encounter.

Problem: Starting any collector fails, Returning Errors

After you add the source, add the collectors, then set up the source at the agent as part of the configuration steps described in Section 3.2, you are ready to start each collector. However, when you attempt to start any of the collectors through the Audit Vault Console, the operation fails with an HTTP error. When this same operation is attempted through the AVCTL `start_collector` command, it again fails, but with the following error message:

    Starting collector...Error executing task start_collector: Cannot find Agent for Collector <source-name>:OSAUD

A quick search of the `agent.err` log file in the Audit Vault Agent home shows the following error:

    SEVERE: java.sql.SQLException: ORA-28150: proxy not authorized to connect as client

The cause stated for an ORA-28150 error is stated as follows: A proxy user attempted to connect as an agent, but the proxy was not authorized to act on behalf of the agent.

Solution:

The solution stated for an ORA-28150 error is as follows: Grant the proxy user permission to perform actions on behalf of the agent by using the SQL `ALTER USER...GRANT CONNECT` command.

One of the configuration steps is to create the Audit Vault source user. Next, you grant proxy connect privilege to the Audit Vault source user through the Agent user. Overlooking this step results in these error messages. This means that the Audit Vault source user has not been granted proxy connect privilege to the Audit Vault source user <avsrcusr> through the agent user <agentusr> to connect to the source database.

See Section 3.2, the second part of Step 2b, about granting proxy connect privilege to the Audit Vault source user <avsrcusr> through the agent user <agentusr>. Performing this step solves the problem. See Example 3-3 for the detailed syntax to perform this step. After the Audit Vault source user is granted this proxy connect privilege, attempts to start any of the collectors should be successful.

Problem: Not sure if the DBAUD_Collector or OSAUD_Collector collectors are collecting from the AUD$ table and the OS file, respectively

After you set up both the DBAUD_Collector and OSAUD_Collector collectors, you want to check to see that they are collecting from the AUD$ table and OS file, respectively.

Solution:

To see if DBAUD_Collector is collecting from the AUD$ table, check the contents of the `DBAUD_Collector_<source-name_prefix><source-id>.log` file in the Audit Vault Agent home `/av/log` directory.

To see if OSAUD_Collector is collecting from the OS File, check the contents of the `orcldb_osaud_<source name>.log` file in the Audit Vault Agent home `/av/log` directory.

Bring each file into an editor and search for entries that indicate that the collector is collecting audit records.

For example, entries like these would be found in the DBAUD_Collector log file:

    ***** Started logging for 'AUD$ Audit Collector' *****
    .
    .
    .
    INFO @ '25/01/2007 19:08:42 -8:00':
         ***** SRC connected OK

    INFO @ '25/01/2007 19:08:53 -8:00':
         ***** SRC data retireved OK
    .
    .
    .

For example, an entry like this would be found in the OSAUD_Collector log file:

    File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
    INFO @ '24/01/2007 18:16:18 -8:00':
    ***** Started logging for 'OS Audit Collector' *****

If everything looks OK, close the editor, then refresh the warehouse using the AVCTL refresh_warehouse command in the Audit Vault Server shell. When this operation completes, log in to the Audit Vault Console as the Audit Vault auditor and examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

Problem: ORA-01017:invalid username/password; logon denied error when starting up the DBAUD_Collector or setting up the REDO_Collector

When you try to start up the DBAUD_Collector or set up the REDO_Collector, you get an ORA-01017: invalid username/password; logon denied error.

Solution:

This error is likely due to a problem with your user name or your password or both in the password file. Try re-creating the user name and password. If the problem persists, re-create the password file.

### 6.3.4 Audit Vault Console

This section describes Audit Vault Console problems that you might encounter.

Problem: Audit Vault Console does not come up in the Web browser

When you try to bring up the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution:

This may be happening because Audit Vault Console is down. To check the status of Audit Vault Console, issue an AVCTL show_av_status command in the Audit Vault Server shell. If the status indicates that the Audit Vault Console is down, issue an AVCTL start_av command in the Audit Vault Server shell to get it started again. Then start up the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the Audit Vault auditor's or administrator's management system.

### 6.3.5 Audit Vault in an Oracle Real Application Clusters (Oracle RAC) Environment

This section describes some problems that you might encounter when you run Audit Vault in an Oracle Real Application Clusters (Oracle RAC) environment.

Problem: In an Oracle RAC environment, the AVCA drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes

When you try to issue an AVCA `add_agent` command from one of the Oracle RAC nodes, the command fails.

Solution:

In an Oracle RAC environment, AVCA commands must be issued from the node on which Oracle Enterprise Manager resides. This is the same node on which the `av.ear` file is deployed.

In an Oracle RAC environment, AVCA and AVCTL commands can be issued only from the node where the `av.ear` file is deployed.

To see where the `av.ear` file is deployed, check to see where the following file is located: `$ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties`

Once you locate the node, run all AVCA and AVCTL commands from that node.

If the node on which the `av.ear` file is deployed is down, deploy the `av.ear` file to another node using the AVCA `deploy_av` command. The command syntax is as follows:

    deploy_av -avadm <usr>/<pwd> -jdbc_str <jdbc connect string>
              -sid <sid> -dbalias <db alias>
              -avconsoleport <av console port>

In this example:

- `-avadm <usr>/<pwd>` is the user name and password of the Audit Vault administrator (user granted AV_ADMIN role). Use a slash (/) to separate the user name and password. The `-avadm` argument can be omitted if the corresponding environment variable, `AVCA_AVADM` is set to `usr/pwd`. If the command-line argument `-usr` is specified, then the command-line argument overrides the environment variable.

- `-jdbc_str <jdbc connect string>` is the JDBC connect string to connect to Audit Vault, which uses the format `jdbc:oracle:<driver type>:@//<host>:<port>/<service name>`.

- `-sid <sid>` is the Oracle system identifier (SID) for the instance.

- `-dbalias <db alias>` is the database alias.

- `-avconsoleport <av console port>` is the port number for the Audit Vault Console.