Troubleshooting WebLogic SAF

This release of WebLogic Server includes the WebLogic Diagnostic Service, which is a monitoring and diagnostic service that runs within the WebLogic Server process and participates in the standard server life cycle. This service enables you to create, collect, analyze, archive, and access diagnostic data generated by a running server and the applications deployed within its containers. This data provides insight into the runtime performance of servers and applications and enables you to isolate and diagnose faults when they occur. WebLogic SAF takes advantage of this service to provide enhanced runtime statistics, notifications sent to imported SAF queues and topics, message life cycle logging JMS messages, and debugging to help you keep your WebLogic domain running smoothly.

Debugging WebLogic SAF

Once you have narrowed the problem down to a specific application, you can activate WebLogic Server’s debugging features to track down the specific problem within the application.

Enabling Debugging

You can enable debugging by setting the appropriate ServerDebug configuration attribute to true. Optionally, you can also set the server StdoutSeverity to Debug.

Enable Debugging Using the Command Line

Enable Debugging Using the WebLogic Server Administration Console

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit (see Use the Change Center).
In the left pane of the console, expand Environment and select Servers.
On the Summary of Servers page, click the server on which you want to enable or disable debugging to open the settings page for that server.
Click Debug.
Expand default.
Select the check box for the debug scope or attributes you want to modify. For example, select the DebugMessaging check box for SAF debugging.
Select Enable to enable (or Disable to disable) the debug scopes or attributes you have checked.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
Not all changes take effect immediately—some require a restart (see Use the Change Center).

This method is dynamic and can be used to enable debugging while the server is running.

Enable Debugging Using the WebLogic Scripting Tool

Use the WebLogic Scripting Tool (WLST) to set the debugging values. For example, the following command runs a program for setting debugging values called debug.py:

The main scope, weblogic, does not appear in the graphic; saf is a sub-scope within weblogic. Note that the fully-qualified DebugScope for DebugSAFSendingAgent is messaging.saf.admin.

Note that you can also use WLST from Java. The following example shows a Java file used to set debugging values:

import weblogic.management.scripting.utils.WLSTInterpreter;
import java.io.*;
import weblogic.jndi.Environment;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

public class test {
	 public static void main(String args[]) {
	try {
		WLSTInterpreter interpreter = null;
		String user="user1";
		String pass="pw12ab";
		String url ="t3://localhost:7001";
		Environment env = new Environment();
		env.setProviderUrl(url);
		env.setSecurityPrincipal(user);
		env.setSecurityCredentials(pass);
		Context ctx = env.getInitialContext();

		interpreter = new WLSTInterpreter();
		interpreter.exec
			("connect('"+user+"','"+pass+"','"+url+"')");
		interpreter.exec("edit()");
		interpreter.exec("startEdit()");
		interpreter.exec
			("cd('Servers/myserver/ServerDebug/myserver')");
		interpreter.exec("set('DebugSAFSendingAgent','true')");	
		interpreter.exec("save()");
		interpreter.exec("activate()");

	} catch (Exception e) {
	System.out.println("Exception "+e);
	}
	}
}

Using the WLST is a dynamic method and can be used to enable debugging while the server is running.

Changes to the config.xml File

Changes in debugging characteristics, through console, or WLST, or command line are persisted in the config.xml file.

This sample config.xml fragment shows a transaction debug scope (set of debug attributes) and a single SAF attribute.

SAF Debugging Scopes

It is possible to see the tree view of the DebugScope definitions using java weblogic.diagnostics.debug.DebugScopeViewer.

You can enable the following registered debugging scopes can be enabled for the WebLogic SAF.

DebugSAFStore (scope weblogic.messaging.saf.store) – prints limited information about SAF’s use of the store.
DebugSAFReceivingAgent (scope weblogic.messaging.saf.receivingagent) – prints information about the SAF receiving side.
DebugSAFSendingAgent (scope weblogic.messaging.saf.sendingagent) – prints information about the SAF sending side.
DebugSAFVerbose (scope weblogic.messaging.saf.verbose) – prints detailed (internal) information.
DebugSAFManager (scope weblogic.messaging.saf.manager) – prints information about SAF management (setting up conversations between agents).
DebugSAFAdmin (scope weblogic.messaging.saf.admin) – prints information about SAF administration (pause/resume).

Note:

For debugging JMS SAF destinations, you can using the following WebLogic JMS scope:

DebugJMSSAF (scope weblogic.jms.saf) – prints information about JMS SAF (store-and-forward) destinations.

For information about WebLogic JMS debugging scopes, see “ Troubleshooting WebLogic JMS” in Configuring and Managing WebLogic JMS.

Request Dyeing

Another option for debugging is to trace the flow of an individual (typically “dyed”) application request through the SAF service. For more information, see “Configuring the Dye Vector via the DyeInjection Monitor” in Configuring and Using the WebLogic Diagnostic Framework.

SAF Message Life Cycle Logging for JMS Messages

The message life cycle is an external view of the events that a JMS message traverses through once it has been stored by a SAF agent and eventually forwarded to a remote JMS destination. Message life cycle logging provides an administrator with easy access to information about the existence and status of stored and forwarded JMS messages from a SAF agent’s viewpoint. In particular, each message log contains information about basic life cycle events such as message storing, forwarding, and expiration. For more information on overall WebLogic logging services, see “ Understanding WebLogic Logging Services” in Configuring Log Files and Filtering Log Messages.

SAF message logging is enabled by default when you create a SAF agent, however, you must specifically enable it on the imported SAF destinations in JMS modules targeted to the SAF agent. Logging can occur on a continuous basis and over a long period of time. It can be also be used in real-time mode while the SAF agent is deployed, or in an off-line fashion when the SAF agent is down. For information about configuring SAF message logging, see:

“ View and configure logs” in the Administration Console Online Help
“ Configure SAF agent JMS message log rotation” in the Administration Console Online Help
“ Configure SAF queue message logging” in the Administration Console Online Help
“ Configure SAF topic message logging” in the Administration Console Online Help

Events in the SAF Message Life Cycle

When SAF message life cycle logging is enabled for an imported SAF queue or topic, a record is added to the SAF agent’s message log file each time a message meets the conditions that correspond to a basic message life cycle event. The life cycle events that trigger a SAF message log entry are as follows:

Stored – This event is logged when a producer sends a message to an imported SAF queue or topic.
Forwarded – This event is logged when SAF forwards a message to a remote queue or topic.
Removed – This event is logged when messages are manually purged from a remote endpoint via a SAF agent.
Expired – This event is logged when a message reaches the expiration time specified on the imported SAF destination.

Message Log Location

where USER_DOMAIN is the root directory of your domain, typically c:\bea\user_projects\domains\USER_DOMAIN, which is parallel to the directory in which WebLogic Server program files are stored, typically c:\bea\wlserver_10.0.

Enabling SAF Message Logging

You can enable or disable SAF message logging for a SAF queue and SAF topic using the WebLogic Server Administration Console. For more information see the following sources.

“ Configure SAF queue message logging” in the Administration Console Online Help
“ Configure SAF topic message logging” in the Administration Console Online Help

WebLogic Java Management Extensions (JMX) enable you to access the SAFAgentRuntimeMBean and SAFAgentRuntimeMBean MBeans to manage SAF message logs. For more information see “ Overview of WebLogic Server Subsystem MBeans” in Programming WebLogic Management Services with JMX

When you enable SAF message logging, you can specify whether the log entry will include all the message header fields or a subset of them; all system-defined message properties or a subset of them; all user-defined properties or a subset of them. You may also choose to include or to exclude the body of the message. For more information about message headers and properties see “ MessageProducer and MessageConsumer” in Programming WebLogic JMS.

SAF Message Log Content

Each record added to the log includes basic information such as the message ID and correlation ID for the subject message. You can also configure the SAF agent to include additional information such as the message type and user properties.

SAF Message Log Record Format

Except where noted, all records added to the SAF Message Life Cycle Log for JMS messages contain the following pieces of information in the order in which they are listed:

Date – The date and time the message log record is generated.
Transaction identifier – The transaction identifier for the transaction with which the message is associated
WLS diagnostic context – A unique identifier for a request or unit of work flowing through the system. It is included in the SAF message log to provide a correlation between events belonging to the same request.
Raw millisecond value for “Date” – To aid in troubleshooting high-traffic applications, the date and time the message log record is generated is displayed in milliseconds.
Raw nanosecond value for “Date” – To aid in troubleshooting high-traffic applications, the date and time the message log record is generated is displayed in nanoseconds.
JMS message ID – The unique identifier assigned to the message.
JMS correlation ID – A user-defined identifier for the message, often used to correlate messages about the same subject.
JMS destination name – The fully-qualified name of the destination server for the message.
SAF message life cycle event name – The name of the message life cycle event that triggered the log entry.
JMS user name – The name of the user who (produced? consumed? received?) the message.
JMS message forwarded identifier – This information is included in the log only when the message life cycle event being logged is the “Message Forwarded” event. If the message forwarded was on a SAF queue, the log will include information about the origin of the remote destination and the OAM identifier for the remote destination known to the SAF agent. If the remote destination is a durable subscriber, the log will also include the client ID for the connection and the subscription name.

The syntax for the message forwarded identifier is as follows:

MC:CA(…):OAMI(wls_server_name.jms.connection#.session#.consumer#)

where

MC stands for message consumer,
CA stands for client address,
OAMI stands for OA&M identifier,
and, when applicable, CC stands for connection consumer.

If the consumer is a durable subscriber the additional information will be shown using the following syntax:

DS:client_id.subscription_name[message consumer identifier]

where DS stands for durable subscriber.

JMS message content – This field can be customized on a per destination basis. However, the message body will not be available.
JMS message selector – This information is included in the log only when the message life cycle event being logged is the “Message Forwarded” event. The log will show the “Selector” argument from the JMS API.

Sample Log File Records

The sample log file records that follow show the type of information that is provided in the log file for each of the message life cycle events. Each record is a fixed length, but the information included will vary depending upon relevance to the event and on whether a valid value exists for each field in the record. The log file records use the following syntax:

Message Stored Event

####<Nov 3, 2005 11:11:02 AM EST> <> <> <1131034262637> <391826> <ID:<864239.1131034262172.0>> <> <SenderSafmodule!safDestinations2!xsafQ2@SendingAgent@D1S1> <Stored> <system> <> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;NON_PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;/mes:JMSExpiration&gt;&lt;mes:JMSPriority&gt;5&lt;/mes:JMSPriority&gt;&lt;mes:JMSRedelivered&gt;false&lt;/mes:JMSRedelivered&gt;&lt;mes:JMSTimestamp&gt;1131034262172&lt;/mes:JMSTimestamp&gt;&lt;mes:Properties/&gt;&lt;/mes:Header&gt;&lt;mes:Body&gt;&lt;mes:Text&gt;EndPointExpireAllWithMessageForward(NP):0&lt;/mes:Text&gt;&lt;/mes:Body&gt;&lt;/mes:WLJMSMessage&gt;> <>

Message Forwarded Event

####<Nov 3, 2005 11:11:30 AM EST> <> <> <1131034290391> <277193> <ID:<864239.1131034288312.0>> <> <SenderSafmodule!safDestinations3!safErrorQueue@SendingAgent@D1S1> <Forwarded> <<WLS Kernel>> <> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;NON_PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;/mes:JMSExpiration&gt;&lt;mes:JMSPriority&gt;5&lt;/mes:JMSPriority&gt;&lt;mes:JMSRedelivered&gt;false&lt;/mes:JMSRedelivered&gt;&lt;mes:JMSTimestamp&gt;1131034288312&lt;/mes:JMSTimestamp&gt;&lt;mes:Properties/&gt;&lt;/mes:Header&gt;&lt;mes:Body&gt;&lt;mes:Text&gt;EndPointExpireAllWithMessageRedirect(NP):0&lt;/mes:Text&gt;&lt;/mes:Body&gt;&lt;/mes:WLJMSMessage&gt;> <>

Message Expired Event

####<Nov 3, 2005 1:04:25 PM EST> <> <> <1131041065929> <42424> <ID:<130865.1131041065828.0>> <> <udd-saf!safDestinations!safRemoteQueue@SendingAgent@D1C1S1> <Expired> <<WLS Kernel>> <> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;1131041065848&lt;/mes:JMSExpiration&gt;&lt;mes:JMSPriority&gt;7&lt;/mes:JMSPriority&gt;&lt;mes:JMSRedelivered&gt;false&lt;/mes:JMSRedelivered&gt;&lt;mes:JMSTimestamp&gt;1131041065828&lt;/mes:JMSTimestamp&gt;&lt;mes:Properties/&gt;&lt;/mes:Header&gt;&lt;mes:Body&gt;&lt;mes:Text&gt;testSAFLogging_Expired&lt;/mes:Text&gt;&lt;/mes:Body&gt;&lt;/mes:WLJMSMessage&gt;> <>

Message Removed Event

####<Nov 3, 2005 11:11:08 AM EST> <> <> <1131034268206> <803337> <ID:<864239.1131034262172.0>> <> <SenderSafmodule!safDestinations2!xsafQ2@SendingAgent@D1S1> <Removed> <<WLS Kernel>> <> <&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"&gt;&lt;mes:Header&gt;&lt;mes:JMSDeliveryMode&gt;NON_PERSISTENT&lt;/mes:JMSDeliveryMode&gt;&lt;mes:JMSExpiration&gt;0&lt;/mes:JMSExpiration&gt;&lt;mes:JMSPriority&gt;5&lt;/mes:JMSPriority&gt;&lt;mes:JMSRedelivered&gt;false&lt;/mes:JMSRedelivered&gt;&lt;mes:JMSTimestamp&gt;1131034262172&lt;/mes:JMSTimestamp&gt;&lt;mes:Properties/&gt;&lt;/mes:Header&gt;&lt;mes:Body&gt;&lt;mes:Text&gt;EndPointExpireAllWithMessageForward(NP):0&lt;/mes:Text&gt;&lt;/mes:Body&gt;&lt;/mes:WLJMSMessage&gt;> <>

Managing SAF Agent Log Files

After you create a SAF agent, you can configure criteria for moving (rotating) old log messages to a separate file. You can also change the default name of the log file.

Rotating Message Log Files

You can choose to rotate old log messages to a new file based on a specific file size or at specified intervals of time. Alternately, you can choose not to rotate old log messages; in this case, all messages will accumulate in a single file and you will have to erase the contents of the file when it becomes too large.

If you choose to rotate old messages whenever the log file reaches a particular size you must specify a minimum file size. After the log file reaches the specified minimum size, the next time the server checks the file size it will rename the current log file and create a new one for storing subsequent messages.

If you choose to rotate old messages at a regular interval, you must specify the time at which the first new message log file is to be created, and then specify the time interval that should pass before that file is renamed and replaced.

Renaming Message Log Files

Rotated log files are numbered in order of creation. For example, the seventh rotated file would be named myserver.log00007. For troubleshooting purposes, it may be useful to change the name of the log file or to include the time and date when the log file is rotated. To do this, you add java.text.SimpleDateFormat variables to the file name. Surround each variable with percentage (%) characters. If you specify a relative pathname when you change the name of the log file, it is interpreted as relative to the server’s root directory.

Limiting the Number of Retained Message Log Files

If you choose to rotate old message log files based on either file size or time interval, you may also wish to limit the number of log files this SAF agent creates for storing old messages. After the server reaches this limit, it deletes the oldest log file and creates a new log file with the latest suffix. If you do not enable this option, the server will create new files indefinitely and you will have to manually clean up these files.

For more information about limiting the number of message log files for SAF agents, see “ Configure SAF agent JMS message log rotation” in the Administration Console Online Help.

Frequently Asked Questions About JMS SAF

This section answers commonly asked questions about how JMS SAF operates in a WebLogic domain.

A. WebLogic Sever’s cluster load balancing is used to pick a sending agent for a given JMS producer. Once a SAF agent is picked, it is used for the life of the JMS producer.

A. A SAF destination can be found the same way a non-SAF JMS destination is found.

JMS clients can look up a SAF Destination in JNDI
createDestination API

JMS clients can also use the createDestination API to find JMS Destination. JMS clients have to use the SAF destination name in a JMS module. The Name must be a fully-qualified name delimited by exclamation points (!). For example:

<EAR Name>!<JMS Module Name>!<ImportedDestinationsName>!<SAFQueue or SAFTopic Name>

Q. Can a JMS producer sending messages to a JMS SAF imported destination be associated with a JMS Unit-of-Order?

A. Yes. For information about the Unit-of-Order feature, see For more information about configuring a Path service, see “ Using Message Unit-of-Order” in Programming WebLogic JMS.

Q. Why does my JMS producer associated with a JMS Unit-of-Order fail to send messages if the sending-side is a cluster?

A. In order to use JMS Unit-of-Order with SAF, you must configure the Path Service for the sending-side cluster. For information about the Path Service feature, see “ Using the WebLogic Path Service” in Configuring and Managing WebLogic JMS.

Q. Do different JMS producers in the same Unit-of-Order pick up the same Sending Agent?

Q. Where do I configure Server Affinity, Load Balancing Enabled, and Forward Delay for a distributed destination that is imported in a sending cluster?

A. Server Affinity and Load Balancing Enabled are configured in the JMS connection factory on which the JMS producer was created. A JMS connection factory creates a JMS connection; a JMS connection creates a JMS session; a JMS session creates a JMS producer. Forward Delay is configured on the JMS distributed destination.

Q. Are the Server Affinity and Load Balancing parameters configured on a JMS connection factory in the sending cluster or server honored on the receiving cluster or server where the JMS Destination resides?

A. Yes. These attributes on the sending cluster or server are honored on the receiving cluster or server. For information about Server Affinity and Load Balancing for distributed destinations, see “ Configuring Distributed Destinations” in Configuring and Managing WebLogic JMS

Q. Do XA transactions on the sending-side of a cluster ever cross the JMS SAF boundary? In other words, can the receiving-side participate with a transaction from the sending- side?

Q. Does JMS SAF preserve the order of messages sent in a JMS session from a sending-side to a JMS destination?

Q. In the SAF Remote Context, should I configure a Principal Name or a Username/Password?

A. You can configure the Remote SAF Context anyway you want. Username and Passwords are stored in the JMS module, and the principal name is stored in a Credential Mapper configured in the sending-side domain.

Q. The membership of my UDD changed and now my SAF does not seem to work properly. What's happening?

A. Make sure all the domains your process communicates with are configured symmetrically with regards to Cross Domain Security. If one of the domains has Cross Domain Security configured, then all must be configured with Cross Domain Security (or be on the appropriate exception lists). See Configuring Domains for Inter-Domain Transactions in Programming WebLogic JTA.

Configuring and Managing WebLogic Store-and-Forward