# Troubleshooting

## Contents
Overview Viewing Enterprise Gateway Trace Files Setting Enterprise Gateway Trace Levels Configuring Enterprise Gateway Trace Files Running Trace at DEBUG level Running Trace at DATA level Integrating Trace Output with Apache log4J Configuring Logging Output Configuring Log Level and Message Getting Help

## Overview
This topic explains how to configure tracing for the Enterprise Gateway, and how to configure logging for message filters to provide an audit trail of message transactions.
## Viewing Enterprise Gateway Trace Files
Each time the Enterprise Gateway starts up, by default, it outputs a trace file to the `INSTALL_DIR\trace` directory. The following example shows an extract from a default Enterprise Gateway trace file:
    INFO    13:58:29:687 [0804] rolling trace to file trace/Oracle Enterprise Gateway.trc started
    INFO    13:58:31:078 [0804] loaded netservice library
    INFO    13:58:31:406 [0804] attempting to connect to entity store at
    federated:file:///C:/enterprisegateway/conf/fed/configs.xml for process Enterprise Gateway
    INFO    13:58:31:984 [0804] Registered EntityTypeFactory: com.vordel.es.EntityTypeFactoryImpl
    ...
The trace file output takes the following format:
    TraceLevel   Timestamp [thread-id] TraceMessage
For example, the first line in the default trace file is described as follows:
Trace Level | `INFO`
Timestamp | `13:58:29:687 (hours:minutes:seconds:milliseconds)`
Thread-id | `0804`
Trace Message | `rolling trace to file trace/Oracle Enterprise Gateway.trc started`
## Setting Enterprise Gateway Trace Levels
The possible trace levels in order of least to most verbose output are as follows: `FATAL` `ERROR` `INFO` `DEBUG` `DATA` where `FATAL` is the least verbose and `DATA` is the most verbose. The default trace level is `INFO`. Setting Trace Levels You can set the trace level in the following different ways:
Startup trace | When the Enterprise Gateway is starting up, it gets its trace level from the `tracelevel` attribute of the `SystemSettings` element in `/system/conf/enterprisegateway.xml`. You can set the trace level in this file if you need to diagnose boot up issues.
System Settings trace | When the Enterprise Gateway has started, it reads its trace level from the System Settings for the Enterprise Gateway process. To set this trace level in the Policy Studio, click the Settings button in the toolbar, select a Trace level from the drop-down list, and click OK.
Interface level trace | You can configure HTTP/HTTPS interfaces with a different trace level from System Settings. For example, the Management Services port (8090) has a default trace level set to `ERROR` to ensure that it is not too verbose at runtime. To set an interface trace level in the Policy Studio, in the Processes tree, right click an HTTP/HTTPS interface port, and select Edit. Select a Trace level from the drop-down list, and click OK.
## Configuring Enterprise Gateway Trace Files
By default, the most recent trace file is named `Oracle Enterprise Gateway.trc`. Older trace files are versioned with the highest version number as oldest (for example, `Oracle Enterprise Gateway0.trc`, `Oracle Enterprise Gateway1.trc`, `Oracle Enterprise Gateway2.trc`, and so on). You can configure the settings for trace file output in the following file: `INSTALL_DIR/system/conf/enterprisegateway.xml`. By default, this file contains the following trace file setting:
    <FileRolloverTrace maxfiles="500" />
This setting means that the Enterprise Gateway writes trace output to `Oracle Enterprise Gateway.trc` in the `trace` directory of the Enterprise Gateway installation. And the maximum number of files that the `trace` directory can contain is `500`. Note: Alternatively, you can configure these trace file settings in `INSTALL_DIR/conf/trace.xml`, which is included by `INSTALL_DIR/system/conf/enterprisegateway.xml`. FileRolloverTrace Attributes The `FileRolloverTrace` element can contain the following attributes:
`filename` | File name used for trace output. Defaults to the `tracecomponent` attribute read from the `SystemSettings` element.
`directory` | Directory where the trace file is written. Defaults to `INSTALL_DIR/trace` when not specified.
`maxlen` | Maximum size of the trace file before it rolls over to a new file. Defaults to `16` MB.
`maxfiles` | Maximum number of files that the `trace` directory contains for this `filename`. Defaults to the maximum integer value (`2147483647`).
`rollDaily` | Whether the trace file is rolled at the start of the day. Defaults to `true`.
Writing Trace Output to Syslog On UNIX and Linux, you can send Enterprise Gateway trace output to `syslog`. In your `INSTALL_DIR/conf/trace.xml` file, add a `SyslogTrace` element, and specify a `facility`. For example:
    <SyslogTrace facility="local0"/>
## Running Trace at DEBUG level
When troubleshooting, it can be useful to set to the trace level to `DEBUG` for more verbose output. When running a trace at `DEBUG` level, the Enterprise Gateway outputs the status of every circuit and filter that it processes into the trace file. Debugging a Filter The trace output for a specific filter takes the following format:
    Filter name {
        Trace for the filter is indented
        to the following point to make it clear
        to identify output from the filter
    } status, in x milliseconds
The status is `0`, `1`, or `2`, depending if the filter failed, succeeded, or aborted. For example, the result of an WS-Security Username Token filter running successfully is as follows:
    DEBUG 12:43:59:093 [11a4] run filter [WS-Security Username Token] {
    DEBUG 12:43:59:093 [11a4]    WsUsernameTokenFilter.invoke: Verify the username and password
    DEBUG 12:43:59:093 [11a4]    WsAuthN.getWSUsernameTokenDetails:
                                 Get token from actor=current actor
    DEBUG 12:43:59:093 [11a4]    Version handler -  creating a new ws block
    DEBUG 12:43:59:108 [11a4]    Version handler - adding the ws element to the wsnodelist
    DEBUG 12:43:59:108 [11a4]    Version handler -  number of ws blocks found:1
    DEBUG 12:43:59:124 [11a4]    No timestamp passed in WS block so no need to check timestamp
    DEBUG 12:43:59:139 [11a4]    WsAuthN.getWSUsernameTokenDetails: Check <Created> element
                                 in token. Value=2010-08-06T11:43:43Z
    DEBUG 12:43:59:139 [11a4]    The WS Nonce TimeStamp Max Size is 1000 and wsNonces cache is 4
    DEBUG 12:43:59:139 [11a4]    Add WS nonce for key [joe:2010-08-06T11:43:43Z].
                                 New cache size [5].
    DEBUG 12:43:59:155 [11a4]    WsBasicAuthN.getUsername: Getting username
    DEBUG 12:43:59:171 [11a4]    WS-Security UsernameToken authN via CLEAR password
    DEBUG 12:43:59:171 [11a4]    VordelRepository.checkCredentials: username=joe
    DEBUG 12:43:59:186 [11a4] } = 1, in 62 milliseconds
Debugging a Circuit The trace output for a circuit shows the circuit running with all its contained filters, and takes the following format:
    Circuit name {
        Filter 1{
            Trace for the filter
        } status, in x milliseconds
        Filter 2{
            Trace for the filter
        } status, in x milliseconds
    }
For example, the following extract shows a circuit called when running the Getting Started demo:
    DEBUG ... run circuit "/axis/services/urn:cominfo"...
    DEBUG ... run filter [Service Handler for 'ComInfoServiceService'] {
    DEBUG ...   Set the service name to be ComInfoServiceService
    DEBUG ...   Web Service context already set to ComInfoServiceService
    DEBUG ...   close content stream
    DEBUG ...   Calling the Operation Processor Chain [1. Request from Client]...
    DEBUG ...   run filter [1. Request from Client] {
    DEBUG ...      run filter [Before Operation-specific Policy] {
    DEBUG ...         run circuit "WS-Security UsernameToken AuthN"...
    DEBUG ...           run filter [WS-Security Username Token] {
                         ...
    DEBUG ...           } = 1, in 62 milliseconds
    DEBUG ...           ..."WS-Security UsernameToken AuthN" complete.
    DEBUG ...       } = 1, in 74 milliseconds
    ...
Debugging at Startup When running a startup trace with a `DEBUG` level set in the `SystemSettings`, the Enterprise Gateway outputs the configuration that it is loading. This can often help to debug any incorrectly configured items at start up, for example:
    DEBUG 14:38:54:206 [1ee0] configure loadable module type RemoteHost, load order = 500000
    DEBUG 14:38:54:206 [1ee0] RemoteHost {
    DEBUG 14:38:54:206 [1ee0]     ESPK: 1035
    DEBUG 14:38:54:206 [1ee0]     ParentPK: 113
    DEBUG 14:38:54:206 [1ee0]     Key Fields:
    DEBUG 14:38:54:206 [1ee0]         name: {csdwmp3308.wellsfargo.com}
    DEBUG 14:38:54:206 [1ee0]         port: {7010}
    DEBUG 14:38:54:221 [1ee0]     Fields:
    DEBUG 14:38:54:221 [1ee0]         maxConnections: {128}
    DEBUG 14:38:54:268 [1ee0]         turnMode: {off}
    DEBUG 14:38:54:268 [1ee0]         inputBufSize: {8192}
    DEBUG 14:38:54:268 [1ee0]         includeContentLengthRequest: {0}
    DEBUG 14:38:54:268 [1ee0]         idletimeout: {15000}
    DEBUG 14:38:54:268 [1ee0]         activetimeout: {30000}
    DEBUG 14:38:54:268 [1ee0]         forceHTTP10: {0}
    DEBUG 14:38:54:268 [1ee0]         turnProtocol: {http}
    DEBUG 14:38:54:268 [1ee0]         includeContentLengthResponse: {0}
    DEBUG 14:38:54:268 [1ee0]         addressCacheTime: {300000}
    DEBUG 14:38:54:268 [1ee0]         outputBufSize: {8192}
    DEBUG 14:38:54:268 [1ee0]         sessionCacheSize: {32}
    DEBUG 14:38:54:268 [1ee0]         _version: {1}
    DEBUG 14:38:54:268 [1ee0]         loadorder: {500000}
    DEBUG 14:38:54:268 [1ee0]         class: {com.vordel.dwe.NativeModule}
    DEBUG 14:38:54:268 [1ee0] }
For details on setting trace levels and running a startup trace, see Setting Enterprise Gateway Trace Levels.
## Running Trace at DATA level
When the trace level is set to `DATA`, the Enterprise Gateway writes the contents of the messages that it receives and sends to the trace file. This enables you to see what messages the Enterprise Gateway has received and sent (for example, to reassemble a received or sent message). Note: On Windows, you can not rely on the console output because it truncates large messages. Every HTTP request handled by the Enterprise Gateway is processed in its own thread, and threads can be reused when an HTTP transaction is complete. You can see what has happened to a message in the Enterprise Gateway by following the trace by thread ID. Because multiple messages can be processed by the Enterprise Gateway at the same time, it is useful to eliminate threads that you are not interested in by searching for items that only match the thread you want. For example, you can do this using vi by specifying the thread ID as a pattern to search for in the trace file. In this case, the thread ID is `145c`:
    :g!/145c/d
The following example shows the `DATA` trace when a message is sent by the Enterprise Gateway (message starts with `snd`):
    DATA 17:45:35:718 [145c]  snd 1495: <POST /axis/services/urn:cominfo HTTP/
         1.1Connection: closeContent-Length: 1295User-Agent: VordelSOAPAction:
         ""Via: 1.0 devsupport2 (Vordel)Host: devsupport2:7070Content-Type:
         text/xml<?xml version="1.0" encoding="UTF-8" standalone="no"?>
         <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
         xmlns:xsd="http://www.w3.org/2001/XMLSchema"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
         <soap:Header>
           <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/
           oasis-200401-wss-wssecurity-secext-1.0.xsd">
           <wsse:UsernameToken xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/
             oasis-200401-wss-wssecurity-utility-1.0.xsd"
             wsu:Id="Id-00000128d05aca81-00000000009d04dc-10">
                  <wsse:Username>joeuser</wsse:Username>
                  <wsse:Nonce EncodingType="utf-8">
                      gmP9GCjoe+YIuK1einlENA==
                  </wsse:Nonce>
                  <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/
                  oasis-200401-wss-username-token-profile-1.0#PasswordText">
                        joepwd
                  </wsse:Password>
                  <wsu:Created>2010-05-25T16:45:30Z</wsu:Created>
             </wsse:UsernameToken>
             </wsse:Security>
          </soap:Header>
          <soap:Body>
              <ns1:getInfo xmlns:ns1="http://stock.samples">
                  <symbol xsi:type="xsd:string">CSCO</symbol>
                  <info xsi:type="xsd:string">address</info>
              </ns1:getInfo>
          </soap:Body>
        </soap:Envelope>
    >
The following example shows the `DATA` trace when a message is received by the Enterprise Gateway (message starts with `rcv`):
    DATA 17:45:35:734 [145c]  rcv 557: <HTTP/1.0 200 OKSet-Cookie: 8Set-Cookie2: 8Content-Type:
         text/xml; charset=utf-8<?xml version="1.0" encoding="UTF-8"?>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
         xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/
         XMLSchema-instance">
         <soapenv:Body>
            <ns1:getInfoResponse soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
            xmlns:ns1="http://stock.samples">
                <getInfoReturn xsi:type="xsd:string">San Jose, CA</getInfoReturn>
            </ns1:getInfoResponse>
          </soapenv:Body>
         </soapenv:Envelope>
    >
If you want to see what has been recieved by the Enterprise Gateway on this thread, run the following command:
    :g!/145c] rcv/d
All `snd` and `rcv` trace statements start and end with < and > respectively. If you are assembling a message by hand from the `DATA` trace, remember to remove characters. In addition, the sending and/or receiving of a single message may span multiple trace statements.
## Integrating Trace Output with Apache log4J
Apache log4j is included on the Enterprise Gateway classpath. This is because some third-party products that the Enterprise Gateway interoperates with require log4j. The configuration for log4j is found in the Enterprise Gateway `INSTALL_DIR/system/lib` directory in the `log4j.properties` file. For example, to specify that the log4j appender sends output to the Enterprise Gateway trace file, add the following setting to your `log4j.properties` file:
    log4j.rootLogger=DEBUG, A1, Vordel
    log4j.appender.Vordel=com.vordel.trace.VordelTraceAppender
## Configuring Logging Output
The Enterprise Gateway provides detailed logging for specific message filters (for example, the request, the time of the request, where the request was routed to, and the response returned to the client). You can configure logging to a number of different locations: Text file XML file Database Local syslog Remote syslog System console To configure where logging information is sent, perform the following steps: In the Policy Studio tree, right-click your process (for example, the Oracle Enterprise Gateway process), and select Logging -> Custom. Specify the required settings on the appropriate tab (for example, Text File, Database, or XML File). Click OK. Click the Deploy button in the toolbar to deploy your settings to the Enterprise Gateway. For details on configuring all the available options, see the Logging Configuration topic.
## Configuring Log Level and Message
You can configure the log level and log message for a specific filter as follows: In the Policy Studio tree, click a policy to display it in the canvas on the right (for example, WS-Security UsernameToken AuthN created in the Getting Started tutorial). Double-click the WS-Security UsernameToken filter on the canvas to edit it. Click Next to display the Log Level and Message screen. Select the Fatal and Failure log levels for troubleshooting. Specify any non-default log messages if required. Click Finish. Click the Deploy button in the toolbar to deploy your settings to the Enterprise Gateway. For more details, see the Log Level and Message topic. Further Details For details on logging the message payload at any point in a circuit, see the Log Payload Filter topic. For details on logging the access details of a message (for example, remote hostname, user login name, and authenticated user name), see the Log Access Filter topic.
## Getting Help
Context-sensitive help is available from the Policy Studio screens. Simply click the Help button on any screen to display the relevant help page for that screen. If you require further information or assistance, please contact the Oracle Support Team. Contacting Support It is important to include as much information as possible when sending support emails to the Oracle Support team. This helps to diagnose and solve the problem in a more efficient manner. The following information should be included with any support query: Name and version of the product (for example, Oracle Enterprise Gateway 11.1.1.5.0). Details of patches that were applied to the product, if any. Platform on which the product is running. A clear (step-by-step) description of the problem or query. If you have encountered an error, the error message should be included in the email. It is also useful to include any relevant trace files from the `/trace` directory of your product installation, preferably with the trace level set to `DEBUG`. Downloading Diagnostics The Enterprise Gateway welcome page available on `http://HOST:8090/` includes a Download Diagnostic Information link. If you need assistance debugging a problem, you can click this link to zip up all the relevant trace and configuration files and send them to the Oracle Support team.