Skip Headers

Oracle® Audit Vault Administrator's Guide Release 10.2.3 Part Number E11059-03 |
Home | Book List | Contents | Index | Contact Us

Previous | Next
| View PDF

# 6 Troubleshooting an Audit Vault System

This chapter provides troubleshooting information for administering an Audit Vault system. This chapter includes the following sections:

- Location of Audit Vault Server Log and Error Files

- Location of Audit Vault Collection Agent Log and Error Files

- Troubleshooting Tips

## 6.1 Location of Audit Vault Server Log and Error Files

Table 6-1 shows the names and a description of the Audit Vault Server log and error files located in the Audit Vault Server `$ORACLE_HOME/av/log` directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur. Log files can be deleted at any time, except for the `avca.log` file, which can only be deleted when the Audit Vault Server is shut down.

Table 6-1 Name and Description of Audit Vault Server Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of errors encountered in collection agent initialization. This file can be deleted at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. This file can be deleted at any time.
`avca.log` | Contains a log of all AVCA commands that have been run and the results of running each command. This file can only be deleted after Audit Vault Server is shut down.
`av_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 10 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. This file can be deleted at any time.
`avorcldb.log` | Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time.
`MSSQLDB-%g.log` | Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. This file can be deleted at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, you must restart av on the Audit Vault Server side (AVCTL stop_av, AVCTL start_av) with the log level set to debug.

If you, as the Audit Vault Administrator, need to do Audit Vault Console troubleshooting, you must enable Enterprise Manager logging. You must modify the following `emomslogging.properties` file in the Audit Vault Server home: `$ORACLE_HOME/sysman/config/emomslogging.properties` on Linux or UNIX systems or `ORACLE_HOME\sysman\config\emomslogging.properties` on Windows systems and add the following lines of information:

    log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
    log4j.appender.avAppender.File=<$ORACLE_HOME>/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log
    log4j.appender.avAppender.Append=true
    log4j.appender.avAppender.MaxFileSize =20000000
    log4j.appender.avAppender.Threshold = DEBUG
    log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
    log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
    log4j.category.oracle =DEBUG, avAppender

This information can be used to debug communication issues between the server and the collection agents.

## 6.2 Location of Audit Vault Collection Agent Log and Error Files

Table 6-2 shows the names and a description of the Audit Vault Collection Agent log and error files located in the Audit Vault Collection Agent `$ORACLE_HOME/av/log` directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur.

Table 6-2 Name and Description of Audit Vault Collection Agent Log and Error Files

File Name | Description
--- | ---
`agent.err` | Contains a log of all errors encountered in collection agent initialization and operation. This file can be deleted at any time.
`agent.out` | Contains a log of all primary collection agent-related operations and activity. This file can be deleted at any time.
`avca.log` | Contains a log of all AVCA commands that have been run and the results of running each command. This file can be deleted at any time.
`avorcldb.log` | Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time.
`<collector-name>_<source-name>_<source-id>.log` | Contains a log of collection operations for the DBAUD, OSAUD, and MSSQLDB collectors. This file can only be deleted after Audit Vault Collection Agent OC4J is shut down. To increase the log level, you must restart the agent OC4J on the Collection Agent side with the appropriate debug level.
`agent_client-``%g``.log.``n` | Contains a log of the collection agent operations and any errors returned from those operations. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 10 MB limit. A concurrent existence of this file is indicated by a `.n` suffix appended to the file type name, such as `av_client-``%g``.log.``n`, where `n` is an integer issued in sequence, for example, `av_client-0.log.1`. This file can be deleted at any time.
`MSSQLDB-%g.log` | Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. This file can be deleted at any time. The `%g` is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, you must restart the Collection Agent OC4J on the Collection Agent side (AVCTL stop_oc4j, AVCTL start_oc4j) with the log level set to debug.
`sqlnet.log` | Contains a log of SQL*Net information.

The directory `Audit_Vault_Agent_Home``/oc4j/j2ee/home/log` contains the logs generated by the collection agent OC4J. In this directory, the file `AVAgent-access.log` contains a log of requests the collection agent receives from the Audit Vault Server. This information can be used to debug communication issues between the server and the collection agent.

Failed configuration commands are located in the Audit Vault Collection Agent `$ORACLE_HOME/cfgtoollogs` directory, which includes the file, configToolFailedCommands. This file contains just the name of the failed command. See the `avca.log` or `avorcldb.log` file for additional information, including any associated errors and error messages.

## 6.3 Troubleshooting Tips

This section describes a number of troubleshooting scenarios that you might encounter with some of the Audit Vault components and how try to resolve each one. The scenarios are placed in the following groupings:

- Audit Vault Server

- Audit Vault Collection Agent

- Audit Vault Collector

- Audit Vault Console

- Audit Vault in an Oracle Real Application Clusters (Oracle RAC) Environment

### 6.3.1 Audit Vault Server

This section describes Audit Vault Server problems that you might encounter.

Problem: Best way to tune Audit Vault Server performance when using the REDO collector.

Following an Audit Vault Server installation, the `streams_pool_size` initialization parameter is set to 150 MB. This parameter must be tuned to maximize REDO collector performance if you are going to be using this collector. In an Oracle Real Application Clusters (Oracle RAC) environment, this parameter must be tuned on all nodes because it is uncertain where the queue will be particularly after an instance startup.

Solution:

Typically, once a REDO collector is configured and started, let it run for a while. This will allow the autotuning feature of Oracle Database to allocate memory for the best database performance for the `streams_pool_size` parameter. Using AWR, check to see if STREAMS AQ has a flow control issue – enqueue being blocked. In the event that you notice that the performance, for example, is only 500 records being applied per second, it may become necessary to manually tune this parameter.

Assuming that you have at least 1 GB of physical memory in your Audit Vault Server system, set this parameter to 200 MB using the SQL command `ALTER SYSTEM SET STREAMS_POOL_SIZE=200;`. Monitor the performance again using AWR. You should achieve a record apply rate of 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB should be sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster. Use the SQL command `ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=av` `n;`, where `n` is the number portion of the SID for each node in the cluster, for example, `av2`, `av3`, `av4`, and so forth, if that is your naming convention.

### 6.3.2 Audit Vault Collection Agent

This section describes Audit Vault Collection Agent problems that you might encounter.

Problem: Audit Vault Agent Status is Blank on the Windows Services Panel

After installing Audit Vault Agent for Windows (32-bit), configuring a source and collectors, then starting the agent on the Audit Vault Server side, a check of the Services Panel on the Windows system where the Audit Vault Agent resides shows that the status is blank, rather than Started.

Solution:

This is normal behavior for the Audit Vault Agent on Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as "Started", however, the Audit Vault Agent is running fine.

You can run the AVCTL `show_agent_status` command to check the status of the Audit Vault Agent. For example, on a Windows system:

    C:\>avctl show_agent_status -agentname agent1
    AVCTL started
    Getting agent metrics...
    --------------------------------
    Agent is running
    --------------------------------
    Metrics retrieved successfully.

Problem: Need to debug a Collection Agent problem

In trying to diagnose an Audit Vault Collection Agent problem, it would be nice to be able to turn on debug logging.

Solution:

This can be done by performing the following set of AVCTL commands on the command-line:

    avctl stop_oc4j
    avctl start_oc4j -loglevel debug

Then check the log output in the Audit Vault Agent `Oracle_home/av/log` directory on Linux and UNIX systems or in the `Oracle_home\av\log` directory on Windows systems.

Turning on debug logging creates more logging and writing overhead, so be sure to turn off debug logging when you are ready to do so by performing the following set of AVCTL commands on the command-line:

    avctl stop_oc4j
    avctl start_oc4j

See the AVCTL stop_oc4j and start_oc4j commands for more information.

Problem: The Agent OC4J or Audit Vault Console OC4J fails to start

After issuing an AVCTLstart_oc4j command, an AVCTL show_oc4j_status command shows that the agent OC4J is not running. Or, after issuing an AVCTL start_av command, an AVCTLshow_av_status command shows that the Audit Vault Console OC4J is not running.

Solution:

Go to `$ORACLE_HOME/av/log/agent.err` log file to see what error message appears in the log.

Or, go to `$ORACLE_HOME/oc4j/j2ee/home` and issue the following command to see what error message appears on the console:

    java -jar oc4j.jar

This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see the following message in the console:

    D:\oracle\product\10.2.3\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar

    08/05/16 10:39:51 Error starting ORMI-Server.  Unable to bind socket: Address already in use: JVM_Bind

Three ports are needed to start the agent OC4J or Audit Vault Console OC4J: RMI, JMS, and HTTP. A port conflict with any of these ports can cause the agent OC4J or Audit Vault Console OC4J to fail to start or the agent service of Audit Vault Console to become unavailable. If there is a port conflict with any of these ports, each of these ports can be modified in the following files at `$ORACLE_HOME/oc4j/j2ee/config` by selecting a port number not in use:

- `rmi.xml`

- `jms.xml`

- `http-web-site.xml or (av-agent-web-site.xml)`

Problem: The setup command returned an error message that the connection to the source database using the credentials in the wallet was not successful

This problem is most likely due to entering an incorrect user name or password or both when issuing the setup command using either the AVORCLDB or AVMSSQLDB command-line utilities.

Solution:

Reissue the setup command again using the correct credentials.

### 6.3.3 Audit Vault Collector

This section describes Audit Vault collector problems that you might encounter.

Problem: Cannot start the DBAUD collector and the log file shows an error

The DBAUD Collector log file (in the Audit Vault Collection Agent home) shows the following entry:

    INFO @ '17/08/2007 15:05:48 02:00':
    Could not call Listener, NS error 12541, 12560, 511, 2, 0

Solution:

In configuring the source and collectors, the last step can be overlooked. This is Step 6 in Section 2.2 about running the `avorcldb setup` command in the Audit Vault Collection Agent home. Overlooking this step, will cause the DBAUD collector from starting.

To verify that this is the problem, set your environment variables for the Audit Vault Collection Agent shell (ORACLE_HOME, PATH, and LD_LIBRARY_PATH).

Change directories to the `network/admin` directory:

Perform the `cat` command on your `tnsnames.ora` file. There should be an entry something like `SRCDB1`. If there is no `SRCDB1` entry in your `tnsnames.ora` file, run the `avorcldb setup` command as shown in Step 6 as previously referenced.

Next, try to connect to the source database with the following command:

    sqlplus /@SRCDB1

If the connection is successful, then your source database is set up correctly. Try starting the DBAUD collector (AVCTL `start_colletor` command).

Problem: Not sure if the DBAUD_Collector or OSAUD_Collector collectors are collecting from the AUD$ table and the OS file, respectively

After you set up both the DBAUD_Collector and OSAUD_Collector collectors, you want to check to see that they are collecting from the AUD$ table and OS file, respectively.

Solution:

To see if DBAUD_Collector is collecting from the AUD$ table, check the contents of the `DBAUD_Collector_<source-name_prefix><source-id>.log` file in the Audit Vault Collection Agent home `/av/log` directory.

To see if OSAUD_Collector is collecting from the OS File, check the contents of the `orcldb_osaud_<source name>.log` file in the Audit Vault Collection Agent home `/av/log` directory.

Bring each file into an editor and search for entries that indicate that the collector is collecting audit records.

For example, entries like these would be found in the DBAUD_Collector log file:

    ***** Started logging for 'AUD$ Audit Collector' *****
    .
    .
    .
    INFO @ '25/01/2007 19:08:42 -8:00':
         ***** SRC connected OK

    INFO @ '25/01/2007 19:08:53 -8:00':
         ***** SRC data retireved OK
    .
    .
    .

For example, an entry like this would be found in the OSAUD_Collector log file:

    File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
    INFO @ '24/01/2007 18:16:18 -8:00':
    ***** Started logging for 'OS Audit Collector' *****

If everything looks OK, close the editor, then refresh the warehouse using the AVCTL refresh_warehouse command in the Audit Vault Server shell. When this operation completes, log in to the Audit Vault Console as the Audit Vault auditor and examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

Problem: ORA-01017:invalid username/password; logon denied error when starting up the DBAUD_Collector or setting up the REDO_Collector

When you try to start up the DBAUD_Collector or set up the REDO_Collector, you get an ORA-01017: invalid username/password; logon denied error.

Solution:

This error is likely due to a problem with your user name or your password or both in the password file as well as a problem with the wallet. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not fix the problem, add the source user to the wallet again using the AVORCLDB setup command. Ensure that it is the same user name and password that you are using on the source database.

Problem: Collector log for MSSQLDB collector indicates a jar file is missing

When the JDBC Driver (`sqljdbc.jar`) for the SQL Server source database, cannot be found in the `Collector Agent home``/jlib` directory, this error will appear in the collector log of the collector being used. Under other circumstances, such as when using the AVMSSQLDB command-line utility, the following error is returned when the JDBC Driver is not located in this directory:

    SQLException3, "JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation."

Solution:

See Step 1 in Section 2.3 for information about the `sqljdbc.jar` (JDBC Driver) for the Microsoft SQL Server source database. Once the JDBC Driver is in place, you must restart the Agent OC4J. See the steps to follow in Stopping and Starting the Agent OC4J in Oracle Audit Vault Collection Agent Installation Guide for more information.

Problem: Unable to connect to source database

When trying to verify either the, ORCLDB or MSSQLDB collector using the verify command, you may be returned the following error "Unable to connect to source database".

Solution:

This error can be returned if the source user you specified in the verify command for either the Oracle Database source or SQL Server source database source database does not have sufficient privileges to connect to the source database. Check and see if the specified source user has sufficient privileges to connect to the respective database. See Step 2 in Section 2.2 for information about creating and granting an Oracle Database source user sufficient privileges to access the database. See Step 2 in Section 2.3 for information about creating and granting a SQL Server database source user sufficient privileges to access the database.

### 6.3.4 Audit Vault Console

This section describes Audit Vault Console problems that you might encounter.

Problem: Audit Vault Console does not come up in the Web browser

When you try to bring up the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution:

This may be happening because Audit Vault Console is down. To check the status of Audit Vault Console, issue an AVCTL show_av_status command in the Audit Vault Server shell. If the status indicates that the Audit Vault Console is down, issue an AVCTL start_av command in the Audit Vault Server shell to get it started again. Then start up the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the Audit Vault auditor's or administrator's management system.

Problem: Need to debug an Audit Vault Console problem

In trying to diagnose an Audit Vault Console problem on the Audit Vault Server, it would be nice to be able to turn on debug logging.

Solution:

This can be done by performing the following set of AVCTL commands on the command-line:

    avctl stop_av
    avctl start_av -loglevel debug

Then check the log output in the Audit Vault Server `Oracle_home/av/log` directory on Linux and UNIX systems or in the `Oracle_home\av\log` directory on Windows systems.

Turning on debug logging will degrade performance, so be sure to turn off debug logging when you are ready to do so by performing the following set of AVCTL commands on the command-line:

    avctl stop_av
    avctl start_av

See the AVCTL stop_av and start_av commands for more information.

### 6.3.5 Audit Vault in an Oracle Real Application Clusters (Oracle RAC) Environment

This section describes some problems that you might encounter when you run Audit Vault in an Oracle Real Application Clusters (Oracle RAC) environment.

Problem: In an Oracle RAC environment, the AVCA drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes

When you try to issue an AVCA `add_agent` command from one of the Oracle RAC nodes, the command fails.

Solution:

In an Oracle RAC environment, AVCA commands must be issued from the node on which Oracle Enterprise Manager resides. This is the same node on which the `av.ear` file is deployed.

In an Oracle RAC environment, AVCA and AVCTL commands can be issued only from the node where the `av.ear` file is deployed.

To see where the `av.ear` file is deployed, check to see where the following file is located: `$ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties`

Once you locate the node, run all AVCA and AVCTL commands from that node.

If the node on which the `av.ear` file is deployed is down, deploy the `av.ear` file to another node using the AVCA `deploy_av` command. The command syntax is as follows:

    deploy_av -sid <sid> -dbalias <db alias>
              -avconsoleport <av console port>

In this example:

- `-sid <sid>` is the Oracle system identifier (SID) for the instance.

- `-dbalias <db alias>` is the database alias.

- `-avconsoleport <av console port>` is the port number for the Audit Vault Console.

Note that when the AVCA `deploy_av` command is issued, a wallet containing the default `avadmin` entries is also created on the other node. However, other entries, such as the source user credentials must be added to the wallet using the `setup` command for the command-line interface (AVORCLDB or AVMSSQLDB) being used that matches the collectors that are in use.

To use the AV Console from this other node, enter its host name or IP address (`<host>`) and port number (`<port>`) as you did previously in the Address field of the browser window (`http://<host>:<port>/av`), but replace the original host name or IP address with that for the other node.