Skip Headers

Oracle® Audit Vault Administrator's Guide Release 10.2.3.1 Part Number E13841-02 |
Home | Book List | Contents | Index | Contact Us

Previous | Next
| View PDF

# A Troubleshooting an Oracle Audit Vault System

This appendix contains:

- Location of Audit Vault Server Log and Error Files

- Location of Audit Vault Collection Agent Log and Error Files

- Troubleshooting Tips

## A.1 Location of Audit Vault Server Log and Error Files

Table A-1 describes the Audit Vault Server log and error files. These files are located in the Audit Vault Server `$ORACLE_HOME/av/log` directory. They contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-1 Name and Description of Audit Vault Server Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of errors encountered in collection agent initialization. You can delete this file at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.
`avca.log` | Contains a log of all `AVCA` and `AVCTL` commands that have been run and the results of running each command. You can only delete this file only after you have shut down the Audit Vault Server.
`av_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence (for example, `av_client-0.log.1`).
`avorcldb.log` | Contains a log of all `AVORCLDB` commands that have been run and the results of running each command. You can delete this file at any time.
`DB2DB-%g.log` | Contains a log of all `AVDB2DB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVDB2DB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug
`MSSQLDB-%g.log` | Contains a log of all `AVMSSQLDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVMSSQLDB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to `debug`, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug
`SYBDB-%g.log` | Contains a log of all `AVSYBDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVSYBDB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to `debug`, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug

If you need to troubleshoot the Audit Vault Console, enable Oracle Enterprise Manager logging. To do so, modify the `emomslogging.properties` file (located in the `ORACLE_HOME``/sysman/config` directory) in the Audit Vault Server home. Add the following lines to this file:

    log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
    log4j.appender.avAppender.File=$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log
    log4j.appender.avAppender.Append=true
    log4j.appender.avAppender.MaxFileSize =20000000
    log4j.appender.avAppender.Threshold = DEBUG
    log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
    log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
    log4j.category.oracle =DEBUG, avAppender

You can use this information to debug communication problems between the server and the collection agents.

## A.2 Location of Audit Vault Collection Agent Log and Error Files

Table A-2 lists the names and a description of the Audit Vault collection agent log and error files located in the `$ORACLE_HOME/av/log` directory. These files contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-2 Name and Description of Audit Vault Collection Agent Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of all errors encountered in collection agent initialization and operation. You can delete this file at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.
`avca.log` | Contains a log of all `AVCA` and `AVCTL` commands that have been run and the results of running each command. You can delete this file at any time.
`avorcldb.log` | Contains a log of all `AVORCLDB` commands that have been run and the results of running each command. You can delete this file at any time.
`DBAUD-and-OSAUD-collector-name_source-name_source-id``.log` | Contains a log of collection operations for the Oracle Database DBAUD and OSAUD collectors. This file has no maximum size limit. To delete this log, shut down the collector, delete the log, and then restart the collector.
`non-Oracle_collector-name``_``source-name``_collector_name-`%`g``.log` | Contains a log of collection operations for the MSSQLDB, SYBDB, and DB2DB collectors. The `%` symbol refers to the generation number of the log file. The maximum log file size is 100 MB. You can only delete this file after you have shut down OC4J. For example, to delete the log where `%``g` is 0, you must stop OC4J. To delete the logs where`%``g` is higher than 0, you can do so while OC4J is running. To increase the log level, restart OC4J on the collection agent side with the appropriate debug level, as in the following example:
    avctl stop_oc4j
    avctl start_oc4j -loglevel error
See Section 7.12 and Section 7.16 for information about these commands, including the available log levels for OC4J.
`agent_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. You can delete this file at any time.
`DB2DB-%g.log` | Contains a log of all `AVDB2DB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVDB2DB` commands, restart OC4J on the collection agent side with the log level set to `debug`, as follows:
    avctl stop_oc4j
    avctl start_oc4j -loglevel debug
`MSSQLDB-%g.log` | Contains a log of all `AVMSSQLDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVMSSQLDB` commands, restart OC4J on the collection agent side with the log level set to debug, as follows:
    avctl stop_oc4j
    avctl start_oc4j -loglevel debug
`SYBDB-%g.log` | Contains a log of all `AVSYBDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%``g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVSYBDB` commands, restart OC4J on the collection agent side with the log level set to `debug`, as follows:
    avctl stop_oc4j
    avctl start_oc4j -loglevel debug
`sqlnet.log` | Contains a log of SQL*Net information.

The Oracle Audit Vault collection agent `$ORACLE_HOME/oc4j/j2ee/home/log` contains the logs generated by the collection agent OC4J. In this directory, the file `AVAgent-access.log` contains a log of requests that the collection agent receives from the Audit Vault Server. Use this information to debug communication problems between the server and the collection agent.

Failed configuration commands are located in the Audit Vault collection agent `$ORACLE_HOME/cfgtoollogs` directory, which includes the file, `configToolFailedCommands`. This file contains only the name of the failed command. See the `avca.log` or `avorcldb.log` file for additional information, including any associated errors and error messages.

## A.3 Troubleshooting Tips

This section contains:

- Checking Trace Files for Detailed Information About Oracle Database Errors

- Troubleshooting Audit Vault Server

- Troubleshooting Audit Vault Collection Agent

- Troubleshooting the Audit Vault Collector

- Troubleshooting Oracle Audit Vault Console

- Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

### A.3.1 Checking Trace Files for Detailed Information About Oracle Database Errors

For detailed information about the cause of an Oracle Database error message, check the trace files. The trace files also indicate problems that may have occurred with the Audit Vault data warehouse, alert, and some configuration issues. The `USER_DUMP_DEST` initialization parameter specifies the current location of the trace files. You can find the value of this parameter by issuing `SHOW PARAMETER USER_DUMP_DEST` in SQL*Plus. See Oracle Database Performance Tuning Guide for more information about trace files

### A.3.2 Troubleshooting Audit Vault Server

Problem: Need to find the best way to tune the Audit Vault Server performance for the REDO collector.

The Audit Vault Server installation process sets the `STREAMS_POOL_SIZE` initialization parameter to 150 MB. If you plan to use the REDO collector, you must tune this parameter to maximize REDO collector performance. In an Oracle Real Application Clusters (Oracle RAC) environment, tune this parameter on all nodes, because it is uncertain where the queue will be after a database instance starts.

Solution:

Typically, after you have configured and started the REDO collector, let it run for a while. This enables the Oracle Database autotuning feature to allocate memory for the best database performance for the `STREAMS_POOL_SIZE` parameter. Using Automatic Workload Repository (AWR), check if Streams AQ has a flow control problem, such as `enqueue` being blocked. If the performance is slow (for example, only 500 records are applied per second), you may need to tune the `STREAMS_POOL_SIZE` parameter.

If you have at least 1 GB of physical memory in your Audit Vault Server system, set the `STREAMS_POOL_SIZE` parameter to 200 MB using the `ALTER SYSTEM` SQL statement, as follows:

    ALTER SYSTEM SET STREAMS_POOL_SIZE=200;

The record apply rate should be 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB is sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster, as follows:

    ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=avn;

Replace `avn` with the SID for each node in the cluster.

### A.3.3 Troubleshooting Audit Vault Collection Agent

Problem: Audit Vault agent status is blank on the Windows Services Panel.

After installing Audit Vault Agent for Microsoft Windows (32-bit), configuring a source and collectors, then starting the agent on the Audit Vault Server side, you notice that the Services Panel on the Windows system where the Audit Vault collection agent resides shows that the status is blank, rather than Started.

Solution:

This is normal behavior for the Audit Vault collection agent on Microsoft Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as Started. However, the Audit Vault collection agent is running without problems.

Run the `avctl` `show_agent_status` command to check the status of the Audit Vault Agent, as follows:

    C:\ORACLE_HOME\agent_dir\bin\avctl show_agent_status -agentname agent1
    AVCTL started
    Getting agent metrics...
    --------------------------------
    Agent is running
    --------------------------------
    Metrics retrieved successfully.

Problem: Need to debug a collection agent problem.

You want to enable debug logging while trying to diagnose an Audit Vault collection agent problem.

Solution:

1. Run the following set of `AVCTL` commands on the command line:

       avctl stop_oc4j
       avctl start_oc4j -loglevel debug

2. Check the log output in the Audit Vault collection agent `$ORACLE_HOME``/av/log` directory.

3. Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

       avctl stop_oc4j
       avctl start_oc4j

See Section 7.12 and Section 7.16 for more information about the `avctl stop_oc4j` and `avctl start_oc4j` commands.

Problem: The Agent OC4J or Audit Vault Console OC4J fails to start.

After you run the `avctl` `start_oc4j` command, an `avctl show_oc4j_status` command shows that OC4J is not running. Or, after you issue `avctl start_av` command, an `avctl show_av_status` command shows that OC4J is not running.

Solution:

Go to `$ORACLE_HOME/av/log/agent.err` log file and check the error message that appears in the log.

Or, go to `$ORACLE_HOME/oc4j/j2ee/home` and issue the following command to find the error message that appears on the console:

    java -jar oc4j.jar

This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see a message similar to the following:

    C:\oracle\product\10.2\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar

    08/05/16 10:39:51 Error starting ORMI-Server.  Unable to bind socket: Address already in use: JVM_Bind

The RMI, JMS, and HTTP ports are necessary for starting OC4J or the Audit Vault Console OC4J. The agent OC4J and Audit Vault Console OC4J can fail to start or the agent service of the Audit Vault Console can become unavailable if these ports have a conflict. If there is a port conflict, you can modify the port settings in the following files at `$ORACLE_HOME/oc4j/j2ee/config` by selecting a port number not in use:

- `rmi.xml`

- `jms.xml`

- `http-web-site.xml or (av-agent-web-site.xml)`

Problem: The setup command returned an error message that the connection to the source database using the credentials in the wallet was not successful.

This problem is most likely due to entering an incorrect user name or password or both when issuing the `setup` command using the `AVORCLDB`, `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB` command-line utility.

Solution:

Reissue the `setup` command again using the correct credentials.

### A.3.4 Troubleshooting the Audit Vault Collector

Problem: Cannot start the DBAUD collector and the log file shows an error.

The DBAUD collector log file (in the Audit Vault collection agent home directory) shows the following entry:

    INFO @ '17/08/2008 15:05:48 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0

Solution:

Ensure that you completed the last step for configuring the source database and collectors, as described in Section 2.3.6, which describes how to run the `avorcldb setup` command in the Audit Vault collection agent home. See also Section 2.2.3 and Section 2.2.4.

Follow these steps:

1. Change directories to the `network/admin` directory:

       $ cd $ORACLE_HOME/network/admin

2. Perform the `cat` command on your `tnsnames.ora` file.

   There should be an entry similar to `SRCDB1`. If there is no `SRCDB1` entry in your `tnsnames.ora` file, then run the `avorcldb setup` command as shown in Section 2.3.6.

3. Try to connect to the source database with the following command, assuming your tnsnames.ora file has an `SRCDB2` entry.

   For example:

       $ sqlplus /@SRCDB1

   If the connection is successful, then your source database is set up correctly.

4. Try starting the DBAUD collector using the `avctl` `start_colletor` command.

   For example:

       $ avctl start_collector -collname REDO_Collector -srcname ORCLSRC1.EXAMPLE.COM

   See Section 7.11 for more information about the `avctl start_collector` command.

Problem: Not sure if the DBAUD and OSAUD collectors are collecting from the AUD$ table and the OS file, respectively.

After you set up both the DBAUD and OSAUD collectors, you want to check that they are collecting from the `AUD$` table and OS file, respectively.

Solution:

To determine if the DBAUD collector is collecting from the `AUD$` table, check the contents of the DBAUD log file, located in the `$ORACLE_HOME/av/log` directory.

To determine if the OSAUD collector is collecting from the OS file, check the contents of the `osaud_``collector-name_source-name_source-id``.log` file in the Audit Vault collection agent- home `$ORACLE_HOME/av/log` directory.

Check each file for entries that indicate that the collector is collecting audit records.

For example, the DBAUD collector log file would have entries similar to the following:

    ***** Started logging for 'AUD$ Audit Collector' *****
    .
    .
    .
    INFO @ '25/10/2008 19:08:42 -8:00':
         ***** SRC connected OK

    INFO @ '25/10/2008 19:08:53 -8:00':
         ***** SRC data retireved OK
    .
    .
    .

The OSAUD collector log file could have an entry as follows:

    File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
    INFO @ '24/10/2008 18:16:18 -8:00':
    ***** Started logging for 'OS Audit Collector' *****

If the log files look correct, then refresh the data warehouse using the `avctl refresh_warehouse` command in the Audit Vault Server shell. When this operation completes, log in to the Audit Vault Console as the Audit Vault auditor. Examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

Problem: ORA-01017:invalid username/password; logon denied error when starting up the DBAUD_Collector or setting up the REDO_Collector.

When you try to start the DBAUD collector or configure the REDO collector, the following error message appears:

    ORA-01017: invalid username/password; logon denied

Solution:

There may be a problem with your user name or password in the password file, or a problem with the wallet credentials. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not correct the problem, add the source user to the wallet again using the `avorcldb setup` command. Ensure that this user is the same user name and password that you are using on the source database.

Problem: Collector log for the MSSQLDB, SYBDB, or DB2DB collector indicates that a jar file is missing.

If the following JDBC driver jar files are missing from the Audit Vault collection agent `$ORACLE_HOME/jlib` library, this error appears in the collector log of the respective collector being used.

- SQL Server: `sqljdbc.jar`

- Sybase ASE: `jconn3.jar`

- IBM DB2: `db2jcc.jar`

Under other circumstances, such as when you use either the `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB` command-line utilities, the following error is appears when the JDBC driver is not in this directory:

    JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation.

Solution:

See the following sections:

- SQL Server: Section 2.4.1 for information about the `sqljdbc.jar` file

- Sybase ASE: Section 2.5.1 for information about the `jconn3.jar` file

- IBM DB2: Section 2.6.1 for information about the `db2jcc.jar` files

After you download and copy these JDBC drivers in place, restart OC4J. See Section 7.16 and Section 7.12 for more information about stopping and starting the OC4J agent.

Problem: Unable to connect to source database.

When you try to verify the ORCLDB, MSSQLDB, SYBDB, or DB2DB collector using the `verify` command, the following error message appears:

    Unable to connect to source database

Solution:

This error appears if the source user that you specified in the `verify` command for the source database does not have sufficient privileges to connect to the source database. Check if the source user has sufficient privileges to connect to the respective database. See the following sections for information about creating a source user with sufficient privileges:

- Section 2.3.2 for Oracle databases

- Section 2.4.2 for Microsoft SQL Server databases

- Section 2.5.2 for Sybase ASE databases

- Section 2.6.2 for IBM DB2 databases

### A.3.5 Troubleshooting Oracle Audit Vault Console

Problem: Audit Vault Console does not appear in the Web browser.

When you try to access the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution:

This may be happening because the Audit Vault Console is down. To check the status of the Audit Vault Console, issue an `avctl show_av_status` command in the Audit Vault Server shell. If the status indicates that the Audit Vault Console is down, issue the `avctl start_av` command in the Audit Vault Server shell to restart it. Then start the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the management system of the Audit Vault auditor administrator.

Problem: Need to debug an Audit Vault Console problem.

You want to enable debug logging while trying to diagnose an Audit Vault Console problem.

Solution:

Run the following commands on the command-line:

    avctl stop_av
    avctl start_av -loglevel debug

Then check the log output in the Audit Vault Server `$ORACLE_HOME/av/log` directory.

Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

    avctl stop_av
    avctl start_av

See Section 7.10 and Section 7.14 for more information about these commands.

### A.3.6 Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

Problem: In an Oracle RAC environment, the avca drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes.

When you try to run the `avca` `add_agent` command from one of the Oracle RAC nodes, the command fails.

Solution:

In an Oracle RAC environment, you must run the `AVCA` commands from the node on which Oracle Enterprise Manager resides. This is the same node on which the `av.ear` file is deployed.

To find where the `av.ear` file is deployed, locate the `$ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties` file is located.

Once you locate the node, run the `AVCA` and `AVCTL` commands from that node.

If the node on which the `av.ear` file is deployed is down, deploy the `av.ear` file to another node using the `avca` `deploy_av` command. See Section 6.4 for more information about this command.

When you run the `avca` `deploy_av` command, on the other node Oracle Database creates a wallet containing the default `avadmin` entries. You must add the other entries, such as the source user credentials, to the wallet by using the `setup` command for the appropriate utility (`AVORCLDB`, `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB`), depending on the collectors being used.

To access the Audit Vault Console from this other node, enter the following URL in the Web browser:

    http://host:port/av

In this specification:

- `host` is the host name or IP address of the other Oracle RAC node

- `port` is the port number for the Oracle RAC node