5/8

# 2 Troubleshooting

The following sections describe how to troubleshoot the mobile server:

- Section 2.1, "Troubleshooting Synchronization"

- Section 2.2, "Troubleshooting the Mobile Server"

- Section 2.3, "Troubleshooting the Mobile Server Repository"

- Section 2.4, "Troubleshooting JVM Errors"

- Section 2.5, "Troubleshooting Security"

## 2.1 Troubleshooting Synchronization

The following sections describe how to troubleshoot the synchronization process or what to do in the event of certain synchronization scenarios:

- Section 2.1.1, "Synchronization Errors and Conflicts"

- Section 2.1.2, "Situations Where the Client is Out of Sync that Triggers a Complete Refresh"

- Section 2.1.3, "The "Inconsistent Datatypes" SQLException Received If Order is Not Correct in Query"

- Section 2.1.4, "MGP Compose Postponed Due to Transaction in the In-Queue"

- Section 2.1.5, "Avoiding the Server Busy Warning"

### 2.1.1 Synchronization Errors and Conflicts

Consult the following sections for details on how to resolve any synchronization errors or conflicts:

- Section 2.1.1.1, "General Synchronization Errors and Conflicts"

- Section 2.1.1.2, "Synchronization Error if Client Device Clock is Inaccurate"

- Section 2.1.1.3, "Synchronization Error After Modifying Client Password"

#### 2.1.1.1 General Synchronization Errors and Conflicts

With the mobile server, you can have the following errors when synchronizing: nullity violations, foreign key constraint violations, or the client updates a row at the same time that the server deletes it.

The mobile server does not automatically resolve synchronization errors. Instead, the mobile server rolls back the corresponding transactions, and moves the transaction operations into the error queue. It is up to the administrator to view the error queue and determine if the correct action occurred. If not, the administrator must correct and re-execute the transaction. If it did execute correctly, then purge the transaction from the error queue.

A mobile server synchronization conflict occurs if:

- Nullity violations.

- Foreign key constraint violations.

- The client and the server update the same row.

- The client and server create rows with the same primary key values.

- The client deletes the same row that the server updates.

- The client updates a row at the same time that the server deletes it.

See Section 2.10, "Resolving Conflict Resolution with Winning Rules" in the Oracle Database Mobile Server Developer's Guide for more information on conflict resolution techniques.

#### 2.1.1.2 Synchronization Error if Client Device Clock is Inaccurate

The client device clock must be accurate within the timezone set on the device before attempting to synchronize. An inaccurate time may result in the following exception during synchronization: `CNS: 9026 "Wrong user name or password. Please enter correct value and reSync."`

#### 2.1.1.3 Synchronization Error After Modifying Client Password

If you have an active client and change its password on the server, then the client cannot synchronize. Return the password back to its original value on the server and retry the synchronization.

### 2.1.2 Situations Where the Client is Out of Sync that Triggers a Complete Refresh

W hen a client is out of sync with the server, any outstanding uploaded transaction from the client is placed in the error queue and a complete refresh is triggered to re-initialize the client data with what is currently on the server.

The following are a list of the situations—ordered from most to least likely—that can trigger a complete refresh for the client:

- Dropping and then republishing the application.

- Synchronizing by the same mobile user from multiple devices on the same platform or from different platforms when the publications are not platform-specific.

- Receiving unexpected server apply phase conditions—such as constraint violations, unresolved conflicts, other database exceptions.

- Modifying the application—such as changing subsetting parameters, or adding or altering publication items.

- Requesting a force refresh from either the server admin or client.

- Two separate applications using the same backend store.

- Unexpected client apply conditions—such as deleting, moving or restoring the client database, database corruption, memory corruption, and other general system failures.

- Loss of transaction integrity between server and client. The server fails post processing after completing a download and disconnecting from the client.

- Data transport corruptions.

### 2.1.3 The "Inconsistent Datatypes" SQLException Received If Order is Not Correct in Query

If you are creating a fast refresh publication item on a table with a composite primary key, the snapshot query should list the primary key columns in the order that they are present in the table definition. This automatically happens during the column selection when MDW is used or when a `SELECT *` query is used. Note that the order of the primary key columns in the table definition may be different from those in the primary key constraint definition.

The following example demonstrates what is valid or invalid given the table definition for TAB1:

    CREATE TABLE TAB1(
     ID1 NUMBER(10) NOT NULL,
     ID2 NUMBER NOT NULL,
     COL1 VARCHAR2(200),
     COL2 VARCHAR2(200),
     ID3 NUMBER(4) NOT NULL);
    ALTER TABLE TAB1 ADD CONSTRAINT TAB1_PK PRIMARY KEY (ID3, ID2, ID1);

The following are valid snapshot queries:

    SELECT * FROM TAB1
    SELECT ID1,ID2,ID3,COL1,COL2 FROM TAB1
    SELECT ID1,ID2,COL1,COL2,ID3 FROM TAB1

The following are invalid snapshot queries:

    SELECT ID3,ID2,ID1,COL1,COL2 FROM TAB1
    SELECT ID3,ID2,COL1,COL2 ID1 FROM TAB1

Define the table columns where the primary key columns appear before other columns. The order of the primary key columns in the table definition order must match the constraint definition in the snapshot query.

### 2.1.4 MGP Compose Postponed Due to Transaction in the In-Queue

If the user synchronized and uploaded some more changes after the last apply cycle for a particular user; by default, the MGP must first apply thse changes before it can compose. If this keeps happening, the compose could be postponed beyond what you would like. By default, the MGP tries to avoid a compose phase postponed due to a transaction in the in-queue by performing an apply for any unprocessed in-queue data before doing a new compose. However, if the MGP compose is postponed due to a transaction in the in-queue, you can modify the following parameters to avoid the error:

- `SKIP_INQ_CHK_BFR_COMPOSE`: By default, this parameter is set to NO. Setting this parameter to YES, then a compose is performed for a client even if there is unprocessed data in the in-queue.

- `DO_APPLY_BFR_COMPOSE`: By default, this parameter is set to YES. If set to YES, the unprocessed data in the in-queue is applied before a client compose. This parameter takes effect only if `SKIP_INQ_CHK_BFR_COMPOSE` is set to NO.

For most situations, preserving the default values for these two parameters avoids the occurrence of the MGP Compose postponed error.

### 2.1.5 Avoiding the Server Busy Warning

The Server Busy warning can be thrown for one of the following reasons:

- When the MGP is processing apply/compose for that user.—To avoid MGP contention with sychronization, MGP should be scheduled to run when few clients are synchronizing. Alternatively, you could use queue-based synchronization, which does not use the MGP at all; thus, avoiding MGP contention with synchronizaiton.

- If a previous synchronization was interrupted for that user and Oracle Database Mobile Server rolls back the transaction—If the Server Busy warning is a result of a long rollback, then Oracle Database recommended tuning steps for rollback operation may reduce the Server Busy state for the client.

## 2.2 Troubleshooting the Mobile Server

The following sections detail how to troubleshoot the mobile server and its repository:

- Section 2.2.1, "Running the Mobile Server With Tracing Enabled"

### 2.2.1 Running the Mobile Server With Tracing Enabled

If you experience any difficulty with the mobile server, you can enable tracing in the mobile server.

To enable tracing in the mobile server, set up your environment as described in Chapter 3, "Tracing and Logging".

## 2.3 Troubleshooting the Mobile Server Repository

The following sections describe how to evaluate, validate and recover the mobile server repository:

- Section 2.3.1, "Troubleshooting the Mobile Server Repository with the Mobile Server Repository Diagnostic Tool"

- Section 2.3.2, "Inspecting Files in the Mobile Repository With the WSH Tool"

- Section 2.3.3, "Modifying IP Address of Machine Where Mobile Repository Exists"

### 2.3.1 Troubleshooting the Mobile Server Repository with the Mobile Server Repository Diagnostic Tool

Customers may modify the mobile server repository in the back-end database and, without realizing it, violate some of the rules. The Mobile Server Repository Diagnostic Tool (MSRDT) provides a mechanism for the customer to analyze, validate, and debug the mobile server repository.

The Mobile Server Repository Diagnosis Tool will automatically correct errors that it can and prints out all results of what was modified incorrectly or missing.

Note:

The output generated is best viewed if you set your column width to 80 or if you pipe it into a file, to view it with Word or WordPad.

- Section 2.3.1.1, "Use the Mobile Server Repository Diagnostic Tool to Validate Your Environment and the Repository"

- Section 2.3.1.2, "Execute the Repository Diagnostics Tool"

#### 2.3.1.1 Use the Mobile Server Repository Diagnostic Tool to Validate Your Environment and the Repository

You can use the Mobile Server Repository and Diagnostic Tool (MSRDT) to validate your environment and what is in the back-end mobile server repository. The following sections describe the validation that occurs:

- Section 2.3.1.1.1, "Validate the Environment for the Mobile Server"

- Section 2.3.1.1.2, "Validate Integrity of Mobile Server Tables and Data"

- Section 2.3.1.1.3, "Validate the Structure and Contents of the Repository"

- Section 2.3.1.1.4, "Validate Application Databases"

##### 2.3.1.1.1 Validate the Environment for the Mobile Server

The MSRDT tool displays the system configuration environment for the host where the mobile server resides, as follows:

- Consolidator version

- Back-end Oracle database version

- Definition of the Java library path

- Definition of the Java `CLASSPATH`

- Operating system architecture, type and version

- Java VM vendor, version, name, home, and info (mode)

- File encoding used

- Path separator and file separator used

- Country, time zone, and user language

- Lists the mobile server administrators

- Lists the contents/current configuration for the `mobile.ora` file

##### 2.3.1.1.2 Validate Integrity of Mobile Server Tables and Data

When the mobile server and the repository are installed, certain tables, constraints, objects, and so on cannot be modified. The MSRDT diagnostic tool checks that these requirements are consistent as with what was installed. This includes the following:

- Required tables exist

- All columns within required tables exist

- Required table attributes exist

- Required constraints exist or have not been modified

- Required sequences exist

- Extra tables have not been added to mobile server schemas

- Application files integrity check: If you have published an application to the server, then check if the folder exists and is not empty. Also, if sharing a repository among multiple mobile servers, ensures that the application is published on this mobile server, where this tool is executed.

##### 2.3.1.1.3 Validate the Structure and Contents of the Repository

If you are experiencing trouble with your repository, execute the MSRDT tool to determine if any of the following have occurred:

- The mapping between primary keys for the map table and corresponding base table must be consistent.

- Every map table must have a corresponding base table.

- Identify the user with the most records in the map table.

- Invalid indexes

- Consistency of the In Queue schema with the ( CPV ) view schema.

- Check if the Error Queue and the In Queue have the same records; that is, both queues should not contain records with the same primary key, which consists of the `CLIENTID`, `TRANID$$`, and `SEQNO$$`.

- The records in the In Queue master table have corresponding records in the In Queue detail table.

- Any DML lock is held by any table in the `C$ALL_LOGGED_TABLES`.

- Invalid triggers.

- Every publication item is included in `C$PUBITEM_PROPS`.

- Validate that all of the users subscribed to a publication also have all of the corresponding sequences assigned to the same publication.

- Invalid sequence values.

- Validate that every record in `C$ALL_SEQUENCES` corresponding `C$WS_``<ID>` window sequence object, where `<ID>` is the values of ID column in `C$ALL_SEQUENCES`.

- Validate that every window sequence has the same id as a record in the `C$ALL_SEQUENCES`.

- Look for any orphaned objects in the repository.

- Verify that the mobile server repository owner is granted with sufficient privileges.

- Validate the MGP properties: If `MGP_SUSPEND(apply suspend)` is false, then `MGP_RUN` must be true.

- Reports on the most recent jobs and their status.

- Check for any automatic synchronization notification sent to a non-existent user in `C$DATA_NOTIFICATION`.

- Check for any automatic synchronization notification that has not been sent by the Device Manager for more than two days.

##### 2.3.1.1.4 Validate Application Databases

You can specify that the application uses a database other than the one in which the mobile server repository resides for all application schema data. If any application databases are defined, then the MSRDT tool checks for the following in all of the defined application databases:

- Check for consistency for the primary keys for both the map table and corresponding base table

- Check for any records that are in the In Queue master table, but not in the corresponding In-Queue detail table

- Check for any tables in the `C$ALL_LOGGED_TABLES` that hold any DML locks

- Check for duplicate records in the Error Queue and the In Queue

- Identify the user with the most records in the map table

- Verify that the In-Queue schema is consistent with the CPV view schema

- Check for any invalid triggers and recompile any invalid triggers

- Check for any invalid application database connections

- Check for any application databases that are registered more than once. This shows up as duplicate records in the `C$DB_INST` table.

- If the publication is dropped from the main database, then the base tables for the publication must also be dropped in the application database. The MSRDT tool checks if the publication was dropped in both the main database and the application database. If the publication tables still exist in the application database, then the MSRDT tool removes the publication entries in `C$ALL_PK_HINTS` and `C$EQ` in the application database.

- Check for any records in the `C$IN_MESSAGES` and `C$INQ` tables in the application database for any dropped clients in the main database. In this case, the MSRDT tool reports the records and moves the `C$IN_MESSAGES` records to the `C$EQ` table, and purges the `C$IN_MESSAGES` table.

#### 2.3.1.2 Execute the Repository Diagnostics Tool

You can use the Mobile Server Repository Diagnostics Tool (`msrdt`) to validate the repository and provide error reporting. It also performs some error recovery.

The following is the usage and syntax for the `msrdt` tool:

    msrdt -v <username>/<password>@<jdbc_url>

Where:

- `<username>/<password>`: The mobile server repository administrator user name and password.

- `<jdbc_url>`: You can specify the JDBC URL of a single Oracle database or an Oracle RAC database, as follows:

  - The URL for a single Oracle database has the following structure: `<host>:<port>:<SID>`

  - The JDBC URL for an Oracle RAC database can have more than one address in it for multiple Oracle databases in the cluster and follows this URL structure:

        jdbc:oracle:thin:@(DESCRIPTION=
         (ADDRESS_LIST=
           (ADDRESS=(PROTOCOL=TCP)(HOST=PRIMARY_NODE_HOSTNAME)(PORT=1521))
           (ADDRESS=(PROTOCOL=TCP)(HOST=SECONDARY_NODE_HOSTNAME)(PORT=1521))
         )
         (CONNECT_DATA=(SERVICE_NAME=DATABASE_SERVICENAME)))

Note:

if you supply a Oracle RAC URL as the JDBC URL, then enclose it within two double-quotes as the operating system treats the equal sign (=) as a delimiter, which truncates the Oracle RAC URL and throws the `syntax error: unexpected token '('.` error

### 2.3.2 Inspecting Files in the Mobile Repository With the WSH Tool

You can use the mobile server shell utility (wsh) to inspect and modify the mobile server repository interactively. For full details, see Appendix B.2, "Running a Script File with the WSH Tool" in the Oracle Database Mobile Server Administration and Deployment Guide.

### 2.3.3 Modifying IP Address of Machine Where Mobile Repository Exists

During the installation, the machine name or IP address is provided by the user where the repository is created. If the IP address of the machine changes, then perform one of the two options:

- If the user provided the machine name; then even after the IP change, the machine name will still work.

- If user provided the IP address—instead of machine name—then after changing the IP address of the repository machine, the user must change the `ADMIN_JDBC_URL` and `THIN_JDBC_URL` parameters in the `mobile.ora` file on the mobile server.

## 2.4 Troubleshooting JVM Errors

This section focuses on how to debug the following Java error:

- Section 2.4.1, "Troubleshooting An Out of Memory Error"

### 2.4.1 Troubleshooting An Out of Memory Error

When you are experiencing the `OutOfMemory` error, then you should have an understanding of JVM memory architecture when tuning mobile server performance.

The following may cause an `OutOfMemory` error:

- A memory leak in the mobile server.

- Not enough physical memory to handle your application.

- In-appropriate allocation of the three memory areas that used by the JVM. See Section 2.4.1.1, "JVM Memory Settings" for a full description.

- Memory being held by the mobile server. See Section 2.4.1.2, "Why is Memory Not Released?" for more information.

- Understanding how threads are consuming your memory. See Section 2.4.1.3, "Thread Memory Consumption and Concurrency" for full details.

#### 2.4.1.1 JVM Memory Settings

JVMs may have different implementations of memory management and garbage collection schemes. But at a higher level, they all arrange the memory in the following three areas:

- Section 2.4.1.1.1, "Java Heap"—The Java heap is where the Java objects live. It is normally the largest of the three.

- Section 2.4.1.1.2, "Permanent Generation"—The memory where the classes are loaded.

- Section 2.4.1.1.3, "Native Space"—The memory used by native code, which includes JVM native code and application JNI calls.

- Section 2.4.1.1.4, "Setting Java Options for Java Memory"—Setting options depending on your environment.

This section describes how to modify the allocation of memory to the JVM memory areas.

Note:

Memory should be allocated properly for the three areas. Otherwise, different kinds of `OutOfMemory` error may surface.

##### 2.4.1.1.1 Java Heap

The Java heap is where Java objects live. It consists of both the young and tenured generations. The amount of Java heap memory that the JVM starts with is designated by the initial heap size option (`-Xms`) and the maximum heap size option (`-Xmx`).

If you see from the stack trace that a Java method throws an `OutOfMemory` error, then you have exhausted your Java heap space.

For example:

    java.lang.OutOfMemoryError: Java heap space

The default settings for the Java heap for a Oracle UNIX JVM is as follows: `Xmx:64M Xms:4M`. However, the default for the mobile server—if you start up the mobile server with the `runmobileserver.bat` executable—is set to `Xmx:256M Xms:512M`.

The size of the space reserved can be specified with the `-Xmx` option. The `-Xms` specifies the space that is immediately committed to the virtual machine. We recommend to allocate ¼ to ½ of the available physical memory to Java Heap. If you set the maximum Java Heap size to be large—such as, 512M—and you still receive this error, then there may be a leak in the Java code.

The amounts specified should be based on the available resources. At the minimum, you should set both values to at least 256 MB. Of course, the amount of memory you allocate depends on what you have available.

Note:

Set the Java heap memory size before starting the mobile server with the `runmobileserver` executable.

##### 2.4.1.1.2 Permanent Generation

The permanent generation holds data needed by the JVM that describes objects which do not have an equivalence at the Java language level. For example, permanent generation is where the classes are loaded. It holds objects that describe classes and methods.

If a classloader method or a String intern method throws an `OutOfMemory` error in the stack trace, then you have run out of permanent generation space.

For example:

    java.lang.OutOfMemoryError: PermGen space at
    java.lang.ClassLoader.defineClass1(Native Method) at
    java.lang.ClassLoader.defineClass(ClassLoader.java:620)

The default for the permanent generation for a Oracle UNIX JVM is 64MB. To set a new initial size for the Oracle JVM, use the `-XX:PermSize` option when starting the virtual machine. To set the maximum permanent generation size use the `-XX:MaxPermSize` option.

##### 2.4.1.1.3 Native Space

The native space is the memory used by native code, which includes JVM native code and application JNI calls. If a native method throws an `OutOfMemory` error or the JVM crashes with such an error, then you run out of native space.

For example:

    java.lang.OutOfMemoryError: requested 14892 bytes. Out of swap space?
    java.lang.OutOfMemoryError: unable to create new native thread

The native space is the (Available physical memory) – (Java heap + Permanent generation). There is no way to set the native space, except to decrease the Java heap or permanent space. If you allocate too much memory for the Java heap, then the native code is left with not enough memory and may run out. If you have to increase the native memory, then decrease the `-Xmx` parameter to a reasonable value to leave enough memory for the native space. If you still get this error, the native code may have a memory leak.

##### 2.4.1.1.4 Setting Java Options for Java Memory

Set the Java options when you start the Java servlet container. The following example sets the initial Java heap to 256M, the maximum Java heap to 512M, and the permanent generation memory to 64M:

    –Xms=256m –Xmx=512m -XX:MaxPermSize=64m

For the mobile server, all modifications for the Java options must be specified on the command-line or in the `runmobileserver.bat` file.

By default, the MGP executes as a job in the Job Scheduler in the mobile server. Thus, the MGP and other mobile server components, such as the Sync Server, share the same memory space. This provides efficiency and manageability; however, if the MGP has a memory leak, then the mobile server is affected. In this case, perform the following:

1. Disable the MGP job.

2. Restart the mobile server.

3. Restart the MGP in a separate JVM with either the `mgp.bat` or, if using UNIX, the `mgp` shell script. This JVM is restarted periodically and may hide the memory leak issue.

Note:

Set the Java memory options for the MGP in the `mgp.bat` file.

With a larger Java heap size, the garbage collector collects less often and consumes less CPU time. Therefore, a larger heap size is desired for better performance. For the mobile server, most of the code is Java code; for the JDBC connection and Java mSync client, most of the code is in native code. So, setting the Java heap size larger, helps the efficiency and performance of the mobile server; however, if it is set too high, the JDBC connection and mSync client may have memory issues.

#### 2.4.1.2 Why is Memory Not Released?

You may expect the mobile server to release the free memory back to operating system after it has finished its work. However, the mobile server holds a large amount of memory even when it is idle. This may not be an indication of memory leak; instead, it may be for one or more of the following reasons:

- If you set the `-Xms` option to a large number—such as, 1024 MB—then you should expect the mobile server process to use at least 1024 MB until the process is killed.

- For performance reasons, the mobile server caches metadata in Java heap memory.

- The garbage collector may not collect the objects right way when they are no longer referenced. In addition, the garbage collector keeps a large amount of free memory in the Java heap for future allocations, instead of returning them to the operating system. You can use Java options to adjust the free memory size; instead, view the mobile server total runtime Java heap size and free heap size in the Mobile Manager at Mobile Manager->Data Synchronization->Host.

#### 2.4.1.3 Thread Memory Consumption and Concurrency

The Java heap and permanent generation together are called managed heap, since the garbage collector manages them. The native space can be divided into native heap and thread stack space. Each thread consumes memory, as follows:

- Each thread created consumes about 1MB stack space, although it is JVM dependent. Take this memory into consideration if you execute multiple threads. For example, on a 32-bit x86 system, the (managed heap + native heap + thread stack size * number of threads) could not exceed 2 GB. On any system, ensure that the total JVM memory is less than the available physical memory size.

- Each thread allocates additional Java and native heap memory as it executes.

- There is an overhead associated with multi-threading. Therefore, be careful when executing too many concurrent threads. If concurrency is set to larger than 20, then you are more likely decreasing the mobile server throughput—instead of increasing it.

You can configure for concurrency with the parameters described in Section 1.2.4, "Configuration Parameters in the MOBILE.ORA that Affect Synchronization Performance".

## 2.5 Troubleshooting Security

The following section describes how to troubleshoot security issues:

- Section 2.5.1, "SSL Certificate Rejection for Client Authentication"

### 2.5.1 SSL Certificate Rejection for Client Authentication

If you are using a reverse proxy and have configured SSL between the client and the reverse proxy, you may receive the following error:

    A certificate is required to complete client authentication.

For all clients you can only use SSL authentication with a signed certificate. If you use a self-signed certificate, you must turn off SSL authentication by adding the following to the `NETWORK` section in the client `devmgr.ini` file:

    DISABLE_SSL_CHECK=YES

This parameter tells the reverse proxy firewall to use SSL encryption for the communication from the client, but not to perform SSL authentication.