20/23

# A Troubleshooting an Oracle Audit Vault System

This appendix contains:

- Location of Audit Vault Server Log and Error Files

- Location of Audit Vault Collection Agent Log and Error Files

- Troubleshooting Tips

## A.1 Location of Audit Vault Server Log and Error Files

Table A-1 describes the Audit Vault Server log and error files. These files are located in the Audit Vault Server `$ORACLE_HOME/av/log` directory. They contain important information about the return status of commands and operations. Use this information to diagnose problems. You should periodically monitor these files, and delete these files to control the amount of disk space the log files use.

Table A-1 Names and Descriptions of Audit Vault Server Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of errors encountered in collection agent initialization. You can delete this file at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.
`alert-email-``%g``.log.``n` | Contains a log of the e-mail alert operations and any errors returned from those operations. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence (for example, `alert-email-0.log.1`).
`alert_troubleticket-``%g``.log.``n` | Contains a log of the trouble ticket alert operations and any errors returned from those operations. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence (for example, `alert-troubleticket-0.log.1`).
`avca.log` | Contains a log of all `AVCA` and `AVCTL` commands that have been run and the results of running each command. You can only delete this file only after you have shut down the Audit Vault Server.
`av_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence (for example, `av_client-0.log.1`).
`avorcldb.log` | Contains a log of all `AVORCLDB` commands that have been run and the results of running each command. You can delete this file at any time.
`DB2DB-%``g``.log` | Contains a log of all `AVDB2DB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVDB2DB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug
`MSSQLDB-%``g``.log` | Contains a log of all `AVMSSQLDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVMSSQLDB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to `debug`, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug
`report-generation-%``g``.log` | Contains a report for both recurring scheduled and manual log messages. You can delete this file at any time. The `%``g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit.
S`YBDB-%``g``.log` | Contains a log of all `AVSYBDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVSYBDB` commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to `debug`, as follows:
    avctl stop_av -loglevel debug
    avctl start_av -loglevel debug
`report-generation-0.log` | Contains report generation-related log messages. This file is created when Oracle Audit Vault generates PDF reports in the Audit Vault Server. You can delete this file at any time.
`policy-0.log` | Contains policy-related log messages for Oracle database audit policies created in Audit Vault. This file is created when Oracle Audit Vault fetches audit settings from a source database, or provisions the audit settings back to the source database. You can delete this file at any time.

If you need to troubleshoot the Audit Vault Console, enable Oracle Enterprise Manager logging. To do so, modify the `emomslogging.properties` file (located in the `$ORACLE_HOME/sysman/config` directory) in the Audit Vault Server home. Add the following lines to this file:

    log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
    log4j.appender.avAppender.File=$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log
    log4j.appender.avAppender.Append=true
    log4j.appender.avAppender.MaxFileSize =20000000
    log4j.appender.avAppender.Threshold = DEBUG
    log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
    log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
    log4j.category.oracle =DEBUG, avAppender

You can use this information to debug communication problems between the server and the collection agents.

## A.2 Location of Audit Vault Collection Agent Log and Error Files

Table A-2 lists the names and a description of the Audit Vault collection agent log and error files located in the `$ORACLE_HOME/av/log` directory. These files contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-2 Names and Descriptions of Audit Vault Collection Agent Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of all errors encountered in collection agent initialization and operation. You can delete this file at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.
`avca.log` | Contains a log of all `AVCA` and `AVCTL` commands that have been run and the results of running each command. You can delete this file at any time.
`avorcldb.log` | Contains a log of all `AVORCLDB` commands that have been run and the results of running each command. You can delete this file at any time.
`collname_srcname_src_id`.`log` | Contains a log of information about what was collected and any collection errors for the DBAUD and OSAUD collectors. This file has no maximum size limit. To delete this log, shut down the collector, delete the log, and then restart the collector. See also Section A.3.4 for more information troubleshooting collectors.
`srcname-collname-#.``log` | Applies to all collectors, as follows: Oracle Database DBAUD, OSAUD, and REDO collectors. Contains monitoring information, such as whether the collector is active and how many records were sent. For the REDO collector, the Streams framework performs the actual collection, so the Oracle Audit Vault agent has no knowledge of the collection. Non-Oracle Database collectors. Contains a log of all collection operations for the MSSQLDB, SYBDB, and DB2 collectors. The `#` symbol refers to the generation number of the log file. The maximum log file size is 100 MB. A `srcname-collname-#``.log.lck` accompanies the log files. You can safely ignore this file. You can only delete this file after you have shut down the agent (for Release 10.2.3.2 collection agents) or OC4J (for Release 10.2.3.1 or earlier agents but not yet upgraded). For example, to delete the log where # is 0, you must stop the agent or OC4J. To delete the logs where # is higher than 0, you can do so while the agent or OC4J is running. To increase the log level, restart the agent or OC4J with the appropriate debug level, as in the following examples: Release 10.2.3.2:
    avctl stop_agent
    avctl start_agent -loglevel error
See Section 7.12 and Section 7.9 for information about these commands, including the available log levels. Release 10.2.3.1 or earlier:
    avctl stop_oc4j
    avctl start_oc4j -loglevel error
See Section 7.15.3 and Section 7.15.2 for information about these commands, including the available log levels.
`agent_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. You can delete this file at any time.
`DB2DB-%g.log` | Contains a log of all `AVDB2DB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVDB2DB` commands, restart the agent (for Release 10.2.3.2 agents) or OC4J (for Release 10.2.3.1 or earlier but not yet upgraded) with the log level set to `debug`, as follows: Release 10.2.3.2:
    avctl stop_agent
    avctl start_agent -loglevel error
See Section 7.12 and Section 7.9 for information about these commands, including the available log levels. Release 10.2.3.1 or earlier:
    avctl stop_oc4j
    avctl start_oc4j -loglevel error
See Section 7.15.3 and Section 7.15.2 for information about these commands, including the available log levels.
`MSSQLDB-%g.log` | Contains a log of all `AVMSSQLDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVMSSQLDB` commands, restart the agent (for Release 10.2.3.2 agents) or OC4J (for Release 10.2.3.1 or earlier but not yet upgraded) on the collection agent side with the log level set to debug, as follows: Release 10.2.3.2:
    avctl stop_agent
    avctl start_agent -loglevel error
See Section 7.12 and Section 7.9 for information about these commands, including the available log levels. Release 10.2.3.1 or earlier (but not yet upgraded):
    avctl stop_oc4j
    avctl start_oc4j -loglevel error
See Section 7.15.3 and Section 7.15.2 for information about these commands, including the available log levels.
`SYBDB-%g.log` | Contains a log of all `AVSYBDB` commands that have been run and the results of running each command. You can delete this file at any time. The `%``g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of `AVSYBDB` commands, restart the agent (for Release 10.2.3.2 agents) or OC4J (for Release 10.3.2.1 or earlier, but not yet upgraded) on the collection agent side with the log level set to `debug`, as follows: Release 10.2.3.2:
    avctl stop_agent
    avctl start_agent -loglevel error
See Section 7.12 and Section 7.9 for information about these commands, including the available log levels. Release 10.2.3.1 or earlier:
    avctl stop_oc4j
    avctl start_oc4j -loglevel error
See Section 7.15.3 and Section 7.15.2 for information about these commands, including the available log levels.
`sqlnet.log` | Contains a log of SQL*Net information.

The Oracle Audit Vault collection agent `$ORACLE_HOME/oc4j/j2ee/home/log` contains the logs generated by the collection agent OC4J. In this directory, the file `AVAgent-access.log` contains a log of requests that the collection agent receives from the Audit Vault Server. Use this information to debug communication problems between the server and the collection agent.

Failed configuration commands are located in the Audit Vault collection agent `$ORACLE_HOME/cfgtoollogs` directory, which includes the file, `configToolFailedCommands`. This file contains only the name of the failed command. See the `avca.log` or `avorcldb.log` file for additional information, including any associated errors and error messages.

## A.3 Troubleshooting Tips

This section contains:

- Checking Trace Files for Detailed Information About Oracle Database Errors

- Troubleshooting Audit Vault Server

- Troubleshooting Audit Vault Collection Agent

- Troubleshooting the Audit Vault Collectors

- Troubleshooting Oracle Audit Vault Console

- Troubleshooting the Oracle Audit Vault Audit Reports

- Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

### A.3.1 Checking Trace Files for Detailed Information About Oracle Database Errors

For detailed information about the cause of an Oracle Database error message, check the trace files. The trace files also indicate problems that may have occurred with the Audit Vault data warehouse, alert, and some configuration issues. The `USER_DUMP_DEST` initialization parameter specifies the current location of the trace files. You can find the value of this parameter by issuing `SHOW PARAMETER USER_DUMP_DEST` in SQL*Plus. See Oracle Database Performance Tuning Guide for more information about trace files

### A.3.2 Troubleshooting Audit Vault Server

This section contains:

- Tuning Audit Vault Server Performance for the REDO Collector

#### A.3.2.1 Tuning Audit Vault Server Performance for the REDO Collector

Problem: You are concerned that the Audit Vault Server performance for the REDO collector is slow. The Audit Vault Server installation process sets the `STREAMS_POOL_SIZE` initialization parameter to 150 MB. If you plan to use the REDO collector, you must tune this parameter to maximize REDO collector performance. In an Oracle Real Application Clusters (Oracle RAC) environment, tune this parameter on all nodes, because it is uncertain where the queue will be after a database instance starts

Solution: Typically, after you have configured and started the REDO collector, let it run for a while. This enables the Oracle Database autotuning feature to allocate memory for the best database performance for the `STREAMS_POOL_SIZE` parameter. Using Automatic Workload Repository (AWR), check if Streams AQ has a flow control problem, such as `enqueue` being blocked. If the performance is slow (for example, only 500 records are applied per second), you may need to tune the `STREAMS_POOL_SIZE` parameter.

If you have at least 1 GB of physical memory in your Audit Vault Server system, set the `STREAMS_POOL_SIZE` parameter to 200 MB using the `ALTER SYSTEM` SQL statement, as follows:

    ALTER SYSTEM SET STREAMS_POOL_SIZE=200;

The record apply rate should be 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB is sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster, as follows:

    ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=avn;

Replace `avn` with the SID for each node in the cluster.

### A.3.3 Troubleshooting Audit Vault Collection Agent

This section contains:

- Blank Status on Windows Services Panel for Audit Vault Agent

- Debugging a Collection Agent Problem

- The Agent OC4J or Audit Vault Console OC4J Failing to Start

- Failed Source Database Connection Due to Invalid Wallet Credentials

#### A.3.3.1 Blank Status on Windows Services Panel for Audit Vault Agent

Problem: After you install Audit Vault Agent for Microsoft Windows (32-bit), configure a source database and collectors, and then start the agent on the Audit Vault Server side, you notice that the Services Panel on the Windows system where the Audit Vault collection agent resides shows that the status is blank, rather than Started.

Solution: This is normal behavior for the Audit Vault collection agent on Microsoft Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as Started. However, the Audit Vault collection agent is running without problems.

Run the `avctl` `show_agent_status` command to check the status of the Audit Vault Agent, as follows:

    C:\ORACLE_HOME\bin\avctl show_oc4j_status

#### A.3.3.2 Debugging a Collection Agent Problem

Problem: You discover that there is a problem with the Audit Vault collection agents, so you want to enable debug logging while trying to diagnose the problem.

Solution:

1. Open a shell or command prompt for the Audit Vault collection agent.

   - UNIX: Set the environment variables, as described in Section 2.2.3.

   - Microsoft Windows: Go to the collection agent `ORACLE_HOME``\bin` directory.

2. Run the following `AVCTL` commands on the command line:

   - For Release 10.2.3.2 agents: Run the following commands:

         avctl stop_agent
         avctl start_agent -loglevel debug

   - For Release 10.2.3.1 or earlier agents that have not been upgraded: Run these commands:

         avctl stop_oc4j
         avctl start_oc4j -loglevel debug

3. Check the log output in the Audit Vault collection agent `$ORACLE_HOME``/av/log` directory.

4. Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it.

   - For Release 10.2.3.2 agents: Run these commands:

         avctl stop_agent
         avctl start_agent -loglevel info

   - For Release 10.2.3.1 or earlier agents that have not been upgraded: Run these commands:

         avctl stop_oc4j
         avctl start_oc4j -loglevel info

#### A.3.3.3 The Agent OC4J or Audit Vault Console OC4J Failing to Start

Problem: After you run the `avctl` `start_oc4j` command, an `avctl show_oc4j_status` command shows that OC4J is not running. Or, after you issue `avctl start_av` command, an `avctl show_av_status` command shows that OC4J is not running.

Solution: Go to `$ORACLE_HOME/av/log/agent.err` log file and check the error message that appears in the log.

Or, go to `$ORACLE_HOME/oc4j/j2ee/home` and issue the following command to find the error message that appears on the console:

    java -jar oc4j.jar

This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see a message similar to the following:

    C:\oracle\product\10.2.3\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar

    09/05/16 10:39:51 Error starting ORMI-Server.  Unable to bind socket: Address already in use: JVM_Bind

The RMI, JMS, and HTTP ports are necessary for starting OC4J or the Audit Vault Console OC4J. The agent OC4J and Audit Vault Console OC4J can fail to start or the agent service of the Audit Vault Console can become unavailable if these ports have a conflict. If there is a port conflict, see Section 4.9 for information about changing the Oracle Audit Vault port number. See also Section 1.3.4.3 for the default collection agent port numbers.

#### A.3.3.4 Failed Source Database Connection Due to Invalid Wallet Credentials

Problem: The `setup` command for the `AVORCLDB`, `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB` command-line utility returns an error message saying that the connection to the source database using the credentials in the wallet was not successful. This problem is most likely due to your entering an incorrect user name or password or both when you issued the `setup` command.

Solution: Reissue the `setup` command again using the correct credentials. See the following sections:

- Section 8.9 for Oracle databases

- Section 9.9 for SQL Server databases

- Section 10.9 for Sybase ASE databases

- IBM DB2 databases: The avdb2db utility has no setup command. For IBM DB2 databases, you only need to change the password of the designated user account.

### A.3.4 Troubleshooting the Audit Vault Collectors

This section contains:

- ORA-01031 Error When You Try to Create a an Oracle Database Collector

- Oracle Source Database DBAUD Log Errors When Starting DBAUD Collector

- DBAUD Collector Does Not Start and the Listener Is Not Available

- Not Sure if the DBAUD and OSAUD Collectors Are Working

- ORA-01017 Error When You Try to Start the DBAUD or REDO Collectors

- MSSQLDB, SYBDB, or DB2 Collector Log Indicates Jar File Is Missing

- Collector Unable to Connect to the Source Database

- Failure of the Computer on Which a Collector Resides

- DB2 Collector Connection Being Denied Due to Lack of License

#### A.3.4.1 ORA-01031 Error When You Try to Create a an Oracle Database Collector

Problem: You are able to add a collector, but the following error appears in the `avorcldb add_collector` command output:

    ERROR: java.sql.SQLException: ORA-01031: insufficient privileges

Afterward, the collector is added, but it fails to collect any audit data.

Solution: The source database user account that you used to create the collector may not have the correct privileges. To remedy this problem:

1. Run the `zarsspriv.sql` script for the source database user account, as described in Section 2.3.1.

2. Restart the collector, as described in Section 2.8.

Another problem can be the compatibility between versions of the Audit Vault Server and source database. See Oracle Audit Vault Installation Guide for your platform for the requirements.

#### A.3.4.2 Oracle Source Database DBAUD Log Errors When Starting DBAUD Collector

Problem: For Oracle source databases that use the DBAUD collector, errors similar to the following may appear in the DBAUD log file when the collectors before the collectors have started:

    INFO @ '12/08/2009 06:50:01 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0
    INFO @ '12/08/2009 06:50:03 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0
    INFO @ '12/08/2009 06:51:03 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0
    INFO @ '12/08/2009 06:51:11 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0
    INFO @ '12/08/2009 06:51:11 02:00':

                 ***** Started logging for 'AUD$ Audit Collector' *****

    INFO @ '12/08/2009 06:51:11 02:00':
         ***** Collector Name = DBAUD_Collector

    INFO @ '12/08/2009 06:51:11 02:00':
         ***** Source Name = HUKSA.EXAMPLE.COM

    INFO @ '12/08/2009 06:51:11 02:00':
         ***** Av Name = AV

    INFO @ '12/08/2009 06:51:11 02:00':
         ***** Initialization done OK

    INFO @ '12/08/2009 06:51:11 02:00':
         ***** Starting CB

    INFO @ '12/08/2009 06:51:13 02:00':
    Getting parameter |AUDAUDIT_DELAY_TIME|, got |20|

    INFO @ '12/08/2009 06:51:13 02:00':
    Getting parameter |AUDAUDIT_SLEEP_TIME|, got |5000|

    INFO @ '12/08/2009 06:51:13 02:00':
    Getting parameter |AUDAUDIT_ACTIVE_SLEEP_TIME|, got |1000|

    INFO @ '12/08/2009 06:51:13 02:00':
    Getting parameter |AUDAUDIT_MAX_PROCESS_RECORDS|, got |1000|

    INFO @ '12/08/2009 06:51:13 02:00':
         ***** CSDK inited OK + 1

    INFO @ '12/08/2009 06:51:13 02:00':
         ***** Src alias = SRCDB1

    INFO @ '12/08/2009 06:51:13 02:00':
         ***** SRC connected OK

    INFO @ '12/08/2009 06:51:13 02:00':
         ***** SRC data retrieved OK

    INFO @ '12/08/2009 06:51:13 02:00':
         ***** Recovery done OK

Solution: You can disregard these error messages. To check that the collector has truly started, you can run the `avctl show_collector_status` command. See Section 7.6.

#### A.3.4.3 DBAUD Collector Does Not Start and the Listener Is Not Available

Problem: You cannot start the DBAUD collector for an Oracle source database, and the listener is not available. The DBAUD collector log file (in the Audit Vault collection agent home directory) shows the following entry:

    INFO @ '17/08/2009 15:05:48 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0

Solution:

Ensure that you completed the last step for configuring the source database and collectors, as described in Section 2.3.5, which describes how to run the `avorcldb setup` command in the Audit Vault collection agent home. See also Section 2.2.3 and Section 2.2.4.

Follow these steps:

1. Change directories to the `network/admin` directory:

       cd $ORACLE_HOME/network/admin

2. Perform the `cat` command on your `tnsnames.ora` file.

   There should be an entry similar to `SRCDB1`. If there is no `SRCDB1` entry in your `tnsnames.ora` file, then run the `avorcldb setup` command as shown in Section 2.3.5.

3. Try to connect to the source database with the following command, assuming your tnsnames.ora file has an `SRCDB2` entry.

   For example:

       sqlplus /@SRCDB1

   If the connection is successful, then your source database is set up correctly.

4. Try starting the DBAUD collector using the `avctl` `start_colletor` command.

   For example:

       avctl start_collector -collname REDO_Collector -srcname ORCLSRC1.EXAMPLE.COM

   See Section 7.11 for more information about the `avctl start_collector` command.

#### A.3.4.4 Not Sure if the DBAUD and OSAUD Collectors Are Working

Problem: You are not sure if the DBAUD collector is collecting from the `SYS.AUD$` table and the OSAUD collector is collecting from the operating system audit file.

Solution: To determine if the DBAUD collector is collecting from the `AUD$` table, check the contents of the DBAUD log file, located in the `$ORACLE_HOME/av/log` directory.

To determine if the OSAUD collector is collecting from the OS file, check the contents of the `osaud_``collector-name_source-name_source-id``.log` file in the Audit Vault collection agent- home `$ORACLE_HOME/av/log` directory.

Check each file for entries that indicate that the collector is collecting audit records.

For example, the DBAUD collector log file would have entries similar to the following:

         ***** Started logging for 'AUD$ Audit Collector' *****
    .
    .
    .
    INFO @ '25/10/2008 19:08:42 -8:00':
         ***** SRC connected OK

    INFO @ '25/10/2008 19:08:53 -8:00':
         ***** SRC data retireved OK
    .
    .
    .

The OSAUD collector log file could have an entry as follows:

    File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
    INFO @ '24/10/2008 18:16:18 -8:00':
    ***** Started logging for 'OS Audit Collector' *****

Log in to the Audit Vault Console as the Audit Vault auditor. Examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

#### A.3.4.5 ORA-01017 Error When You Try to Start the DBAUD or REDO Collectors

Problem: When you try to start the DBAUD collector or configure the REDO collector, the following error message appears:

    ORA-01017: invalid username/password; logon denied

Solution: Ensure that the host that you specified for the `-av` argument of the `avorcldb add_collector` command is correct. You can confirm the host name by checking the `tnsnames.ora` file for the source database.

Alternatively, there may be a problem with your user name or password in the password file, or a problem with the wallet credentials. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not correct the problem, then add the source user to the wallet again using the `avorcldb setup` command. Ensure that this user is the same user name and password that you are using on the source database.

#### A.3.4.6 MSSQLDB, SYBDB, or DB2 Collector Log Indicates Jar File Is Missing

Problem: The collector log for the MSSQLDB, SYBDB, or DB2 collector indicates that a jar file is missing. If the following JDBC driver jar files are missing from the Audit Vault collection agent `$ORACLE_HOME/jlib` library, then this error message appears in the collector log of the respective collector being used.

- SQL Server: `sqljdbc.jar`

- Sybase ASE: `jconn3.jar`

- IBM DB2: `db2jcc.jar`

Under other circumstances, such as when you use either the `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB` command-line utilities, the following error appears when the JDBC driver is not in this directory:

    JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation.

Solution: See the following sections:

- SQL Server: Section 2.4.1 for information about the `sqljdbc.jar` file

- Sybase ASE: Section 2.5.1 for information about the `jconn3.jar` file

- IBM DB2: Section 2.6.1 for information about the `db2jcc.jar` file

After you download and copy these JDBC drivers in place, restart the collection agent. See Section 7.12 and Section 7.9 for more information about stopping and starting the agent for Release 10.2.32 agents. For agents that were created in Release 10.2.3.1 or earlier, restart OC4J by running the `avctl stop_oc4j` and `avctl start_oc4j` commands.

#### A.3.4.7 Collector Unable to Connect to the Source Database

Problem: When you try to verify the ORCLDB, MSSQLDB, SYBDB, or DB2 collector using the `verify` command, the following error message appears:

    Unable to connect to source database

Solution: This error appears if the source user that you specified in the `verify` command for the source database does not have sufficient privileges to connect to the source database. Check if the source user has sufficient privileges to connect to the respective database. See the following sections for information about creating a source user with sufficient privileges:

- Section 2.3.1 for Oracle databases

- Section 2.4.2 for Microsoft SQL Server databases

- Section 2.5.2 for Sybase ASE databases

- Section 2.6.2 for IBM DB2 databases

#### A.3.4.8 Failure of the Computer on Which a Collector Resides

Problem: The computer on which you have created a collector fails.

Solution: For collectors that retrieve audit data from the database (and not operating system files), you can collect audit trails from another host computer by moving the collector to a different computer. To do so, see the following sections:

- Section 8.4, for the `avorcldb alter_collector` command; applies to the DBAUD collector only

- Section 9.4, for the `avmssqldb alter_collector` command; applies to the server-side trace logs only

- Section 10.4, for the `avsybdb alter_collector` command

#### A.3.4.9 DB2 Collector Connection Being Denied Due to Lack of License

Problem: When you run the `avdb2db verify` command or perform other DB2 collector-related functions, the log file may report that the connection was denied because the license is missing. This can result from having the wrong version of the IBM Data Server Driver for JDBC and SQLJ installed. You must have version 3.50 or later.

Solution: Check the version of the IBM Data Server Driver for JDBC and SQLJ driver, and if necessary, upgrade it.

To check the version of this driver, run the following command on the `db2jcc.jar` file:

    java -cp jar_file_directory_path/db2jcc.jar com.ibm.db2.jcc.DB2Jcc -version

If the version of the driver is earlier than version 3.50, then follow the instructions in Section 2.6.1 to upgrade to the correct version.

### A.3.5 Troubleshooting Oracle Audit Vault Console

This section contains:

- Audit Vault Console Not Appearing in the Web Browser

- Audit Vault Console Problem Requiring Debugging

- Oracle RAC Node Containing the Audit Vault Console Becomes Disabled

#### A.3.5.1 Audit Vault Console Not Appearing in the Web Browser

Problem: When you try to access the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution: This may be happening because the Audit Vault Console is down. To check the status of the Audit Vault Console, issue an `avctl show_av_status` command in the Audit Vault Server shell or command prompt. If the status indicates that the Audit Vault Console is down, issue the `avctl start_av` command in the Audit Vault Server shell or command prompt to restart it. Then start the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the management system of the Audit Vault auditor administrator.

#### A.3.5.2 Audit Vault Console Problem Requiring Debugging

Problem: You want to enable debug logging while trying to diagnose an Audit Vault Console problem.

Solution: Run the following commands on the command-line:

    avctl stop_av
    avctl start_av -loglevel debug

Then check the log output in the Audit Vault Server `$ORACLE_HOME/av/log` directory.

Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

    avctl stop_av
    avctl start_av

See Section 7.10 and Section 7.13 for more information about these commands.

#### A.3.5.3 Oracle RAC Node Containing the Audit Vault Console Becomes Disabled

Problem: In an Oracle RAC environment, the node on which the Audit Vault Console is installed becomes disabled. When this node becomes unavailable, the Console does not automatically fail over to another node as the repository database does. As a result, the Audit Vault Console application is no longer available to users.

Solution: You must manually bring up the Audit Vault Console on another node in the Oracle RAC cluster. See Section 4.6 for more information.

### A.3.6 Troubleshooting the Oracle Audit Vault Audit Reports

This section contains:

- Oracle Audit Vault Reports Not Displaying

- Oracle Audit Vault Reports Not Showing Any Data

- Not Sure if Audit Data Is Appearing in the Data Warehouse

- Advanced Alerts Unable to Fire and New Alerts Cannot Be Created

#### A.3.6.1 Oracle Audit Vault Reports Not Displaying

Problem: When you select the Reports tab in the Audit Vault Console, an error message appears saying that the report cannot be displayed.

This problem can occur in the following situations:

- The Oracle Audit Vault installation may not have completed successfully. Check the Audit Vault Server installation log files, which are located in the `$ORACLE_HOME/av/log` directory. See "Location of Audit Vault Server Log and Error Files" for more information.

- The database cannot register the Oracle listener XDB HTTP endpoint. Check the Audit Vault Server database listener status:

      lsnrctl status

  The output should contain the following line of text:

      ...(PORT=5707 ))(Presentation=HTTP)(Session=RAW))

  If it does not, then follow these steps:

  1. Log in to SQL*Plus as the `XDB` user or a user who has been granted the `EXECUTE` privilege for the `DBMS_XDB` PL/SQL package.

         sqlplus xdb_admin
         Enter password: password

  2. Run the following procedure:

         SQL> EXEC DBMS_XDB.SETHTTPPORT(5707);

  3. Connect as user `SYS` with the `SYSDBA` privilege, and then run the following `ALTER SYSTEM` statement:

         SQL> ALTER SYSTEM REGISTER;

  Alternatively, follow these steps:

  1. Disable Oracle Database Vault.

     See Oracle Database Vault Administrator's Guide for instructions on disabling (and re-enabling) Oracle Database Vault.

  2. Log in to SQL*Plus as user `SYS` with the `SYSDBA` privilege, and then run the following SQL statements:

         SQL> EXEC DBMS_XDB.SETHTTPPORT(5707);
         SQL> ALTER SYSTEM REGISTER;

  3. Re-enable Oracle Database Vault.

#### A.3.6.2 Oracle Audit Vault Reports Not Showing Any Data

Problem: When you generate a report, the Oracle Audit Vault reports are not showing any data, even in cases where the audit trail data can be retrieved from source database.

Solution: A report filter may be incorrectly set, or the collection agent or collectors may not be running. Try the following solutions:

- Check the filters for the report. For example, the report may have been enabled to show only events that occurred in the last 24 hours. Disable this filter and then check that the warehouse data is being refreshed. See Oracle Audit Vault Auditor's Guide for more information.

- If the audit trail data cannot be retrieved, ensure that the collectors are enabled. See Section 2.7 and Section 2.8 for more information.

#### A.3.6.3 Not Sure if Audit Data Is Appearing in the Data Warehouse

Problem: Even though the Oracle Audit Vault data warehouse is refreshed automatically with audit data, you are not sure if the data warehouse is capturing this data.

Solution: If you believe that the audit data is not being updated in the data warehouse, then check the Activity Overview Report, described in Oracle Audit Vault Auditor's Guide. If you find that data is missing, then check the server-side log files (alert and trace logs, located in the `$ORACLE_HOME/av/log` directory) for errors. If there are errors, then contact Oracle Support.

#### A.3.6.4 Advanced Alerts Unable to Fire and New Alerts Cannot Be Created

Problem: An advanced alert that uses a user-defined function is no longer able to be fired. Furthermore, you find that you cannot add any new alerts for the affected Audit Vault alert rule set, which is comprised of the category and source database type. For example, you would no longer be able to create or use alerts for the Account Management category of the Oracle Database source type. In fact, none of the alerts for the affected rule set may work. This problem can occur if the function has been changed or dropped, or the privileges for running the function have been changed. As a result, the alert is no longer valid.

Solution: Check that the function has not been modified or its privileges changed. Ensure that the `AVREPORTUSER` account has been granted the `EXECUTE` privilege for the function. Then recreate the alert to use the corrected function. Afterward, the alert and other alerts in the rule set should work correctly.

### A.3.7 Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

This section contains:

- avca drop_agent Command Failing

#### A.3.7.1 avca drop_agent Command Failing

Problem: In an Oracle RAC environment, the avca drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes.

When you try to run the `avca` `add_agent` command from one of the Oracle RAC nodes, the command fails.

Solution: In an Oracle RAC environment, you must run the `AVCA` commands from the node on which Oracle Enterprise Manager resides. This is the same node on which the `av.ear` file is deployed.

To find where the `av.ear` file is deployed, locate the `$ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties` file is located.

Once you locate the node, run the `AVCA` and `AVCTL` commands from that node.

If the node on which the `av.ear` file is deployed is down, deploy the `av.ear` file to another node using the `avca` `deploy_av` command. See Section 6.6 for more information about this command.

When you run the `avca` `deploy_av` command, on the other node Oracle Database creates a wallet containing the default `avadmin` entries. You must add the other entries, such as the source user credentials, to the wallet by using the `setup` command for the appropriate utility (`AVORCLDB`, `AVMSSQLDB`, `AVSYBDB`, or `AVDB2DB`), depending on the collectors being used.

To access the Audit Vault Console from this other node, enter the following URL in the Web browser:

    http://host:port/av

In this specification:

- `host` is the host name or IP address of the other Oracle RAC node

- `port` is the port number for the Oracle RAC node