Oracle Berkeley DB Java Edition 11G R2 Change Log

Library Version 11.2.4.0, Release 4.0.117

Changes in 4.0.117

General Changes

  1. Fixed a bug that causes a EnvironmentFailureException LOG_FILE_NOT_FOUND when using a particular sequence of operations with a Database configured for duplicates (DatabaseConfig.setSortedDuplicates(true)) or a DPL secondary key with a MANY_TO_ONE or MANY_TO_MANY relationship. The sequence of operations that causes the bug is:
    1. A single record exists (no duplicates) for key A
    2. New transaction T1 starts
    3. Key A is deleted in T1
    4. Two or more duplicates are inserted for key A in T1
    5. The T1 transaction aborts
    Later, the log file containing the original version of the record is cleaned and deleted, causing an exception when that record is subsequently accessed. An example stack trace is below. 
    (JE 4.0.92) var/db/bdb_userNode fetchTarget of 0x3151/0x44b2638 parent
IN=729921304 IN class=com.sleepycat.je.tree.DBIN
lastFullVersion=0x3189/0x13616b0 parent.getDirty()=false state=0
LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is
invalid and must be closed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1300)
...
    Thanks to "helg" on OTN for reporting this and working with us to diagnose the problem. [#17252]

  2. Added documentation for the je.rep.node.priority configuration property. [#17684]

  3. Fixed a bug which could cause PreloadStatus.EXCEEDED_TIME to be incorrectly returned from the Database.preload() method. [#18577]

  4. Added the property EnvironmentConfig.CLEANER_LAZY_MIGRATION to provide finer control over log cleaner, checkpointing and eviction behavior. See the javadoc for details. [#18650]

  5. Fixed a bug that causes a fatal exception during recovery (opening of a JE Environment) under certain conditions. An example of the exception stack trace is below. 
    Exception in thread "main" com.sleepycat.je.EnvironmentFailureException:
(JE 4.0.92) last LSN=0x424/0xe4fc1 LOG_INTEGRITY: Log information is incorrect,
problem is likely persistent. Environment is invalid and must be closed.
    at com.sleepycat.je.recovery.RecoveryManager.traceAndThrowException(RecoveryManager.java:3052)
    at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:867)
    at com.sleepycat.je.recovery.RecoveryManager.buildINs(RecoveryManager.java:621)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:513)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:175)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:529)
    at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:204)
    at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:230)
    at com.sleepycat.je.Environment.(Environment.java:212)
    at com.sleepycat.je.Environment.(Environment.java:166)
    ...
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.92)
fetchTarget of 0x89/0x4c2e29 parent IN=4883580 IN
class=com.sleepycat.je.tree.DIN lastFullVersion=0x424/0xdcec4
parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is
likely invalid. Environment is invalid and must be closed.
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
    at com.sleepycat.je.tree.DIN.fetchTarget(DIN.java:520)
    at com.sleepycat.je.tree.IN.findParent(IN.java:2704)
    at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:879)
    at com.sleepycat.je.recovery.RecoveryManager.replayINDelete(RecoveryManager.java:1770)
    at com.sleepycat.je.recovery.RecoveryManager.replayOneIN(RecoveryManager.java:945)
    at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:846)
    ... 11 more
Caused by: java.io.FileNotFoundException: 00000089.jdb (The system cannot find
the file specified)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Unknown Source)
    at java.io.RandomAccessFile.(Unknown Source)
    at com.sleepycat.je.log.FileManager$1.(FileManager.java:993)
    at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:992)
    at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:888)
    at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
    at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
    ... 17 more

    The bug only occurred under the following conditions:

    No data loss occurs as a result of this bug. By using a version of JE with the bug fix, such environments can be opened and used normally. Thanks to OTN user justindthomas for reporting the problem and working with us to diagnose and fix it. [#18663]


  6. Fixed a bug that causes an IllegalStateException when a DPL Converter mutation is used for class evolution, and Replicas are upgraded first (as prescribed) in a replication group. [#18690]

  7. Fixed a bug which can make log utilization inaccurate and prevent log cleaning, when Environment.removeDatabase or Environment.truncateDatabase is called, and the program crashes or exits before any other information is written to disk. For example, if the Environment is closed normally after the removal/truncation, or a scheduled checkpoint occurs, then the bug will not occur. [#18696]

  8. All public JE exception and statistics classes should be serializable. Some were not, and have been fixed. The following classes are now certified to be serializable. [#18738]


  9. Fixed a bug which caused a NullPointerException under certain circumstances when key prefixing is enabled. This was originally reported on the OTN Forum. [#18773]

  10. Fixed a bug that caused log cleaner threads to hang with the following stack trace. This could occur infrequently in applications which use EnvironmentMutableConfig.setExceptionListener() and have a high level of concurrent log cleaning. [#18791] 
       java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:374)
        at java.util.HashSet.add(HashSet.java:200)
        at com.sleepycat.je.dbi.EnvironmentImpl.registerExceptionListenerUser(EnvironmentImpl.java:730)
        at com.sleepycat.je.utilint.DaemonThread.(DaemonThread.java:60)
        at com.sleepycat.je.cleaner.FileProcessor.(FileProcessor.java:110)
        at com.sleepycat.je.cleaner.Cleaner.doClean(Cleaner.java:461)
        at com.sleepycat.je.dbi.EnvironmentImpl.invokeCleaner(EnvironmentImpl.java:1879)
        at com.sleepycat.je.Environment.cleanLog(Environment.java:1559)

  11. Fixed a problem where Database.count() could return the wrong value for replicated databases with duplicates, after the environment had experienced a replication rollback. [#18816]

  12. Fixed a problem which was intermittent and seen only on the master node of a replication group which using synchronous transaction commits. The application would see the following stack trace. The problem was transient, no data was lost, and the system could re-open successfully. [#18882] 
    java.io.FileNotFoundException: ... (The system cannot find the file specified)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Unknown Source)
    at java.io.RandomAccessFile.(Unknown Source)
    at com.sleepycat.je.log.FileManager$1.(FileManager.java:998)
    at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:998)
    at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:893)
    at com.sleepycat.je.rep.stream.FeederReader$SwitchWindow.fillNext(FeederReader.java:523)
    at com.sleepycat.je.log.FileReader.readData(FileReader.java:758)
    at com.sleepycat.je.log.FileReader.readNextEntryAllowExceptions(FileReader.java:259)
    at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:230)
    at com.sleepycat.je.rep.stream.FeederReader.scanForwards(FeederReader.java:284)
    at com.sleepycat.je.rep.stream.MasterFeederSource.getWireRecord(MasterFeederSource.java:62)
    at com.sleepycat.je.rep.impl.node.Feeder$OutputThread.run(Feeder.java:659)

  13. When a disk corruption occurs that overwrites a log entry with non-zero data, a checksum error is now properly reported via an EnvironmentFailureException, and tools such as DbScavenger and DbVerifyLog work properly. Earlier, if a corruption overwrote specific parts of the log entry, another exception was mistakenly throw, such as: 
    java.lang.ArrayIndexOutOfBoundsException at
java.util.zip.Adler32.update(Adler32.java:47) at
com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:59) at
com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:55) at
com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:928)
...
    Thanks to Jean-Christophe on OTN for reporting this bug.

  14. Opening a com.sleepycat.bind.serial.StoredClassCatalog on a replica node in a JE HA replication group incorrectly resulted in a ReplicaWriteException. This has been fixed. Thanks to user13067358 on OTN for reporting this problem. [#18938]

  15. A DPL class evolution bug caused an exception or incorrect behavior when renaming a field, when the name change caused the alphabetical order of fields in the class to change. An example of such an exception follows. However, other types of exceptions could also occur. 
    java.lang.IllegalArgumentException: Can not set java.lang.Integer field 
com.sleepycat.persist.test.EvolveClasses$RenameSecField.new_secKey2 to java.lang.String
at java.lang.reflect.Field.set(Field.java:657)
at com.sleepycat.persist.impl.ReflectionAccessor$ObjectAccess.read(ReflectionAccessor.java:422)
at com.sleepycat.persist.impl.ReflectionAccessor.readSecKeyFields(ReflectionAccessor.java:253)
at com.sleepycat.persist.impl.ComplexFormat$EvolveReader.readObject(ComplexFormat.java:2127)
at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(PersistEntityBinding.java:115)
at com.sleepycat.persist.impl.PersistEntityBinding.entryToObjectInternal(PersistEntityBinding.java:83)
at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(PersistEntityBinding.java:64)
at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:597)
at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:584)
...
    This has been fixed. [#18961]

  16. Fixed a bug that causes Cursor.getNextDup or EntityCursor.nextDup to advance to a following key under certain rare conditions. This also has a side effect of causing Database.delete to delete duplicate records for the following key under the same conditions. The conditions that lead to the bug are:
    1. A Database is configured for duplicates or a DPL secondary key with a MANY_TO_XXX relationship is used.
    2. The environment or transaction is explicitly configured for Serializable Isolation (this is not the default isolation mode).
    3. One or more records for key A exist.
    4. The key following key A, key B, has two or more duplicates, and the first duplicate for key B is deleted.
    5. Timing and conditions are such that the deleted duplicate for key B has not been internally compressed out of the Btree.
    6. Cursor.getNextDup or EntityCursor.nextDup is called when positioned on the last record for key A, or Database.delete is called for key A. In this case, the record(s) with key B will mistakenly be returned or deleted.
    [#19026]

  17. Fixed a bug that causes a EnvironmentFailureException LOG_FILE_NOT_FOUND when using CacheMode.EVICT_BIN. The bug did not cause data loss, but did prevent recovery and opening of the Environment. Heavy eviction during a checkpoint increases the likelihood that the problem will manifest itself. [#19422]

Changes in 4.0.103

Features

  1. Added the LocalCBVLSN field for replication group members to the information displayed by com.sleepycat.je.rep.util.DbGroupAdmin -dumpGroup or through the JMX MBean dumpReplicationState() operation. The LocalCBVLSN field is an indicator of the progress a replicated node has made in executing replicated operations.

    Users can monitor the LocalCBVLSN values for group members as a way of gauging if the group is executing at an even pace. The values need not be identical, because they are only update periodically, but a large or growing discrepancy indicates that some group members are lagging.[#18430]


  2. Several new com.sleepycat.je.CacheMode enumeration values have been added which can help to improve cache memory usage and reduce Java GC costs for some applications:

    The CacheMode is configurable using new database and environment properties:

    Note that the MAKE_COLD mode has been enhanced since its introduction in JE 3.3. In JE 4.0, this mode will perform explicit eviction when the JE cache is full, and blocking between threads may be reduced.

    See the CacheMode Javadoc for more information. [#18410]


  3. Several new statistics have been added com.sleepycat.je.EnvironmentStats to support the new CacheMode functionality and to make it easier to understand caching behavior. [#18512]


  4. Two new properties: com.sleepycat.je.rep.ReplicationConfig.REPLICA_TIMEOUT and com.sleepycat.je.rep.ReplicationConfig.FEEDER_TIMEOUT have been defined to permit application level control of connection timeouts. (4.0.100)

General Changes

  1. Fixed a bug which can cause Database.count() to return incorrect values when an environment is opened read-only. [#18180]

  2. JE applications run on the Resin application server could experience a NullPointerException at startup. The problem occurs because Resin's implementation of java.util.logging.Loggers.getHandlers() can return null instead of an empty array. A check for a null return was added. The problem was originally reported on the OTN forum. [#18277]

  3. Opening a read-only environment with a non default value for the com.sleepycat.je.util.FileHandler.level property resulted in a NullPointerException. In addition, output to the je.info file was disabled for read only environments. This has been fixed. [#18334]

  4. Fixed a bug in Database.preload() which would cause a latch on a database root node to be held in the face of the cache filling or preload taking too much time. [#18396]

  5. Fixed a bug where the cache usage was not correctly calculated when using the -XX:+UseCompressedOops JVM option. [#18462]

  6. Replicated applications running with the property je.cleaner.threads set to a value greater than 1, where the application does not fit in cache, can experience a deadlock which shows the following stacktraces. This has been fixed. [#18475] 
    "Cleaner-1":
at com.sleepycat.je.cleaner.UtilizationProfile.putFileSummary(UtilizationProfile.java:846)
- waiting to lock <0xf00db530> (a com.sleepycat.je.cleaner.UtilizationProfile)
at com.sleepycat.je.cleaner.UtilizationProfile.flushFileSummary(UtilizationProfile.java:835)
at com.sleepycat.je.cleaner.UtilizationTracker.evictMemory(UtilizationTracker.java:95)
at com.sleepycat.je.dbi.EnvironmentImpl.specialEviction(EnvironmentImpl.java:2379)
at com.sleepycat.je.evictor.PrivateEvictor.startBatch(PrivateEvictor.java:96)
at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:306)
at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:245)
- locked <0xf0222028> (a com.sleepycat.je.evictor.PrivateEvictor)
at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:273)
at com.sleepycat.je.dbi.EnvironmentImpl.criticalEviction(EnvironmentImpl.java:2360)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:502)

  ...

Cleaner-4:
at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:229)
- waiting to lock <0xf0222028> (a com.sleepycat.je.evictor.PrivateEvictor)
at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:273)
at com.sleepycat.je.dbi.EnvironmentImpl.criticalEviction(EnvironmentImpl.java:2360)
at com.sleepycat.je.dbi.CursorImpl.criticalEviction(CursorImpl.java:295)
at com.sleepycat.je.Cursor.beginMoveCursor(Cursor.java:2704)
  ...
at com.sleepycat.je.rep.vlsn.VLSNIndex.getLTEFileNumber(VLSNIndex.java:675)
at com.sleepycat.je.rep.impl.node.GlobalCBVLSN.getCleanerBarrierFile(GlobalCBVLSN.java:100)
at com.sleepycat.je.rep.impl.node.RepNode.getCleanerBarrierFile(RepNode.java:1287)
at com.sleepycat.je.rep.impl.RepImpl.getCleanerBarrierStartFile(RepImpl.java:1156)
at com.sleepycat.je.cleaner.UtilizationProfile.getBestFileForCleaning(UtilizationProfile.java:289)
- locked <0xf00db530> (a com.sleepycat.je.cleaner.UtilizationProfile)
at com.sleepycat.je.cleaner.FileSelector.selectFileForCleaning(FileSelector.java:225)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:202)
- locked <0xf0035cd0> (a com.sleepycat.je.cleaner.FileProcessor)
at
	    com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:140)

  7. Replicated applications with a high level of cache eviction can experience a deadlock where the checkpointer thread displays the following type of stacktrace. This has been fixed. [#18475] 
    "Checkpointer" daemon prio=3 tid=0x08518000 nid=0x1d waiting on condition [0xedca8000]
- parking to wait for  <0xf0211680> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at com.sleepycat.je.tree.IN.latch(IN.java:332)
          ...
at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:245)
- locked <0xf01809c0> (a com.sleepycat.je.evictor.PrivateEvictor)
          ...
at com.sleepycat.je.rep.vlsn.VLSNTracker.flushToDatabase(VLSNTracker.java:366)
- locked <0xf018e510> (a com.sleepycat.je.rep.vlsn.VLSNTracker)
          ...
at com.sleepycat.je.rep.impl.RepImpl.preCheckpointEndFlush(RepImpl.java:656)
at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:490)

  8. Fixed a bug which caused an IllegalArgumentException to be thrown when parsing various JE parameters if the default language was not English (e.g.if it was set to Turkish). Thanks to "Mike" on OTN for reporting this. [#18504]

  9. Fixed a bug in DbCacheSize that caused a slight overestimation of memory used by the Btree. This utility now prints out detailed memory usage for each level in the Btree. [#18520]

  10. A replicated environment could very rarely experience this exception, which was due to an incorrect assertion check. The assertion has been fixed. [#18524] 
    com.sleepycat.je.EnvironmentFailureException: (JE 4.0.97) Locker:
com.sleepycat.je.txn.BasicLocker UNEXPECTED_STATE: Unexpected internal
state, may have side effects.

at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:376)
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:262)
at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:134)
at com.sleepycat.je.txn.Locker.nonBlockingLock(Locker.java:487)
at com.sleepycat.je.tree.MapLN.isEvictable(MapLN.java:174)
at com.sleepycat.je.tree.BIN.evictInternal(BIN.java:1004)
at com.sleepycat.je.tree.BIN.evictLNs(BIN.java:965)
at com.sleepycat.je.evictor.Evictor.evictIN(Evictor.java:820)
   ...

  11. Fixed a bug where a Replica could fail to detect that it had lost its connection to the Master when its network cable was unplugged. As a result, an election would not be initiated by the Replica. [#18554]

  12. The directions in com.sleepycat.je.rep.RollbackProhibitedException and com.sleepycat.je.util.DbTruncateLog were not clear enough about how to handle a RollbackProhibitedException, and have been improved. DbTruncateLog itself had a bug which made it quietly skip the log truncation. [#18558].

  13. Fixed a bug that causes an internal RefreshException to be thrown when calling com.sleepycat.persist.EntityStore.getPrimaryIndex. This occurs only under these conditions: [#18594] (4.0.101)

  14. Fixed a bug that causes an NullPointerException to be thrown when calling com.sleepycat.persist.EntityStore.getPrimaryIndex. This occurs only under these conditions: [#18594] (4.0.102)

  15. Fix a bug that causes the following exception under rare conditions when calling Database.sync for a deferred write database. 
     java.lang.IndexOutOfBoundsException
 at com.sleepycat.bind.tuple.TupleInput.readBoolean(TupleInput.java:186)
 at com.sleepycat.je.cleaner.PackedObsoleteInfo.countObsoleteInfo(PackedObsoleteInfo.java:60)
 at com.sleepycat.je.log.LogManager.serialLogInternal(LogManager.java:671)
 at com.sleepycat.je.log.SyncedLogManager.serialLog(SyncedLogManager.java:40)
 at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:388)
 at com.sleepycat.je.recovery.Checkpointer.logSiblings(Checkpointer.java:1285)
    [#18567] (4.0.102)

Changes in 4.0.92

Features

  1. Normal operation of JE HA requires that at least a simple majority of electable nodes be available to form a quorum for election of a new Master, or when committing a transaction with default durability requirements. The number of electable nodes (the Electable Group Size) is obtained from persistent internal metadata that is stored in the environment and replicated across all members. See Replication Group Life Cycle for details.

    Under exceptional circumstances, a simple majority of nodes may become unavailable for some period of time. With only a minority of nodes available, the overall availability of the group can be adversely affected.

    To deal with this exceptional circumstance, especially if the situation is likely to persist for an unacceptably long period of time, a mechanism has been added by which you can modify the way in which the number of electable nodes, and consequently the quorum requirements for elections and commit acknowledgments, is calculated. See ReplicationMutableConfig.ELECTABLE_GROUP_SIZE_OVERRIDE and Replication Guide appendix, Managing a Failure of the Majority [#18022]


  2. A number of improvements have been made to the JE Dynamic MBeans. [#17913], [#18194]

API Changes

  1. Added getGroupName() to com.sleepycat.je.rep.util.ReplicationGroupAdmin, which returns the group name. ReplicationGroupAdmin.getMasterSocket() was inadvertently public and has been made private. [#17841]

  2. The following methods in com.sleepycat.je.rep.ReplicatedEnvironmentStats:
    getProtocolAvgReadNanos()
    getProtocolAvgWriteNanos()
    getProtocolReadNanos()
    getProtocolWriteNanos()
    were intended to provide throughput information for a replicated JE environment, but required some computation from the user. They have been replaced with methods that present throughput in a more forthright manner:
    getProtocolBytesReadRate()
    getProtocolBytesWriteRate()
    getProtocolMessageReadRate()
    getProtocolMessageWriteRate()
    [#17913]

  3. A replication group can rollback a committed transaction in rare circumstances, if there is network partitioning or there are master failures when transactions have been committed with the type of durability that requires no replica acknowledgments. In such cases, a node which was previously a master may have transactions committed to disk which were not propagated to the group and which are not contained by the current master. By default, JE handles this case transparently. A new property has been added to limit the number of transactions that can be automatically rolled back, as a safeguard. See com.sleepycat.je.rep.ReplicationConfig.TXN_ROLLBACK_LIMIT and the new com.sleepycat.je.rep.RollbackProhibitedException. [#17926]

  4. Several new classes and methods, and a bug fix were added to make it easier to track replication group changes when using the com.sleepycat.je.rep.monitor package.

    com.sleepycat.je.rep.monitor.GroupChangeEvent was being generated whenever an electable node joined a replication group. Instead, this event should only be generated when the replication group composition changes.

    The class com.sleepycat.je.rep.monitor.GroupChangeEvent now defines a new enumeration: GroupChangeType and a new method GroupChangeEvent.getChangeType(). Together these changes make it possible to identify the type of structural change (whether a node was added or removed) and the affected node.

    The class com.sleepycat.je.rep.monitor.MonitorChangeListener now defines two new notify methods: MonitorChangeListener.notify(JoinGroupEvent) and MonitorChangeListener.notify(LeaveGroupEvent). These new methods, along with the new events: com.sleepycat.je.rep.monitor.JoinGroupEvent and com.sleepycat.je.rep.monitor.LeaveGroupEvent, permit the listener to track the dynamic state of electable nodes as they join and leave the replication group. You will need to supply implementations for the new notify methods if you have an existing class that implements the MonitorChangeListener interface.

    This problem was originally reported here on the OTN forum. [#18006]


  5. It is our intent to let the JE user set java.util.logging properties programmatically. JE logging output is most easily specified by setting the com.sleepycat.je.util.ConsoleHandler.level and com.sleepycat.je.util.FileHandler.level through the standard java.util.logging configuration file or MBean, but the java.util.logging API does not directly support setting handler levels. JE now accepts the properties com.sleepycat.je.util.ConsoleHandler.level and com.sleepycat.je.util.FileHandler.level as properties to the je.properties file or the EnvironmentConfig.setConfigParam() method to support programmatic setting of these values. See the Logging chapter for more information. [#18017]

  6. The maximum value for log file size (je.log.fileMax) on the Android platform was increased from 1MB to 10MB.

General Changes

  1. Database.preload(PreloadConfig) no longer throws NullPointerException if null is passed for an argument. Instead, it uses a default PreloadConfig. [#17784] (4.0.72)

  2. Fixed a bug on Android that appears as a NullPointerException when using a SerialBinding. A workaround was added for a Harmony (Dalvik) bug that occurs when the ObjectInputStream.readClassDescriptor is overridden. [#17903]

  3. In certain cases, the JE MBeans did not return exception information correctly. For example, calling getDatabaseStats() operation with an invalid database name would return this message:
    java.rmi.UnmarshalException: Error unmarshaling return; nested exception is java.lang.ClassNotFoundException: com.sleepycat.je.DatabaseNotFoundException ....
    instead of a proper message explaining the problem. This has been fixed. [#17913]

  4. Fixed an inefficiency which resulted in extra computation on all transactional operations when there are multiple lock tables. The inefficiency became noticeable when using hundreds of lock tables. [#17944]

  5. Fixed a performance issue during highly concurrent read operations by hundreds of threads, caused by contention on a java.util.concurrent.locks.ReentrantReadWriteLock taken during construction of a ThreadLocal. The ThreadLocal is only used during stats collection, and the implementation now makes initialization lazy and conditional. [#17944]

  6. Fixed a problem with Database.close when the EnvironmentConfig.ENV_IS_LOCKING property is set to false. This caused Environment.truncateDatabase, removeDatabase and renameDatabase to throw LockTimeoutException when the database is opened and closed prior to performing the truncate, remove or rename operation. It also caused a resource/memory leak (a single lock was not released by Database.close) that could impact applications that frequently open and close databases. Again, this only impacts applications that have explicitly set EnvironmentConfig.ENV_IS_LOCKING to false. [#17985]

  7. Fixed a bug that caused PreloadStats.getNLNsLoaded to return a non-zero number, even when PreloadConfig.setLoadLNs(false) was specified. Note that no LNs were loaded -- only the returned stat was incorrect. [#18021]

  8. A minor improvement was made to JE cache management. The calculated amount of memory will be slightly more accurate, especially for applications that write with random key values. The change could, in principle, prevent an OutOfMemoryError, but the improvement is small enough that it is unlikely to have a noticeable impact on most applications. [#18035]

  9. Fixed an issue on Android where ClassNotFoundException could sometimes be returned. Possible cases are when a custom key comparator, or a SerialBinding is used. The problem was caused by a bug in the Harmony (Dalvik) implementation of Class.forName() when used with the thread's context class loader. [#18163]

  10. Fixed a bug on Android where a full Checkpoint or Environment.sync() might cause later errors to appear. This problem was caused by a bug in the Harmony (Dalvik) implementation of java.util.IdentityHashMap which is described in http://issues.apache.org/jira/browse/HARMONY-6419. [#18167]

  11. The JConsole plugin would exit if an non-numeric value was entered in the graphing interval text box. This has been corrected to instead return an exception message. [#18194]

  12. Fixed a problem in JE HA where in the unusual case of a node transitioning from a Master to a Replica state, the node could sometimes fail with an com.sleepycat.je.EnvironmentFailureException as it tried to establish a replication stream with the new master. A sample stack trace is enclosed below: 
    Replica unexpected exception com.sleepycat.je.EnvironmentFailureException: ...
com.sleepycat.je.EnvironmentFailureException: (JE 4.0.80)
node4(3):/tmp/scaleDir4/env Rep stream not sequential.
Current VLSN: 94,338,473 next log entry VLSN: 94,338,515
UNEXPECTED_STATE: Unexpected internal state, may have side effects.
 at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:342)
 at com.sleepycat.je.rep.impl.node.Replica.doRunReplicaLoopInternalWork(Replica.java:433)
 at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:353)
 at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:295)
 at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:914)
    [#18212]

  13. Fixed a problem where cleaned and deleted log files could accumulate in the log cleaner's backlog, or list of files to be cleaned. This occurs when multiple cleaner threads are configured. The impacts of this problem are:
    1. The EnvironmentStats.getCleanerBacklog stat is incorrect, which could lead the application to unnecessarily increase the number of cleaner threads.
    3. If EnvironmentConfig.CLEANER_MAX_BATCH_FILES is set to a non-zero value, log cleaning is disabled when the number of deleted files in the backlog reaches this limit.
    [#18179]

  14. Fixed a bug that can cause LogFileNotFound (loss of log information) when using temporary databases (DatabaseConfig.setTemporary(true) or StoreConfig.setTemporary(true)). This is known to occur only under heavy concurrent cache eviction and log cleaning. [#18227]

Utility Changes:

  1. Added a utility named com.sleepycat.je.rep.util.DbGroupAdmin which presents the functionality of the com.sleepycat.je.rep.util.ReplicationGroupAdmin class in a command line utility. It lets the user both display the composition of a replication group and remove nodes from the group. DbGroupAdmin should be used as part of JE HA configuration checklist. [#17841]

  2. DbCacheSize now supports -measure to measure the actual amount of cache size used, without the use of -data. When -data is not specified, -measure will cause only the keys and internal nodes to be kept in cache. This allows measuring the amount of memory necessary to hold the internal nodes for a data set, without allocating enough memory to hold the entire data set, including leaf nodes. [#18036]

Examples:

  1. Added a new example to illustrate the use of RMI for request forwarding in a replicated system. See the javadoc for je.rep.quote.StockQuotesRMIForwarding.

Changes in JE 4.0.71

File Format
New Features
API
Performance and General Changes
Direct Persistence Layer
Utility Changes
Documentation, Installation and Integration

Log File On-Disk Format Changes:

JE 4.0.117 has moved to on-disk file format 7.

The change is forward compatible in that JE files created with release 3.3 and earlier can be read when opened with JE 4.0.117. The change is not backward compatible in that files created with JE 4.0 cannot be read by earlier releases. Note that if an existing environment is opened read/write, a new log file is written by JE 4.0 and the environment can no longer be read by earlier releases.

Changes in API and exception handling create a binary incompatibility between JE 4.0 and JE 3.X. Users are advised to recompile their applications when moving to JE 4.0.

New Features:

  1. This release features JE High Availability, which permits the use of JE with replication. See the High Availability Guide for an introduction to the product, and the com.sleepycat.je.rep javadoc for API specifications.

  2. JE now provides two JMX MBean implementations for monitoring a running JE application. A JConsole plugin is also provided for accessing the new JEMonitor and RepJEMonitor classes. Environment statistics can be collected and graphed, and other state information can be perused. See the how-to for more information.

  3. It is critical that invalid files are not added to a backup set, since then both the live environment and the backup will be invalid. Two new classes now support verification of log files prior to making a backup.

    The com.sleepycat.je.util.LogVerificationInputStream class enables log verification programmatically as files are being copied. The com.sleepycat.je.util.DbVerifyLog class is a simple command line utility for inclusion in backup scripts. [#16476]


  4. Cursor and Database record retrieval has been optimized to significantly reduce I/O when DatabaseEntry.setPartial is used to suppress return of the data item and the READ_UNCOMMITTED isolation mode is used. See "Using Partial DatabaseEntry Parameters"in the Cursor javadoc for more information. [#16859]

  5. The EnvironmentConfig.setNodeName(String nodeName) and EnvironmentConfig.getNodeName() methods have been added to match those same methods in ReplicationConfig. Setting the EnvironmentConfig node name will cause java.util.logging messages and thread names to include the user-specified name. This was previously only true for replicated environments, and has been extended to work for non-replicated environments, as a debugging aid when using multiple environments in the same JVM. The feature was originally requested on the OTN forum. [#17629] (4.0.70)

API Changes:

  1. The close() method may now be called repeatedly on Database, Cursor, Environment, JoinCursor, and EntityStore instances, and abort() may be called repeatedly on Transaction instances. Whereas they used to throw an exception if close() or abort() was called multiple times, they now return silently on subsequent calls. [#16214]

  2. XAEnvironment.getXATransaction() has been removed from the Javadoc. This is an internal entrypoint and should not be used by general users. getXATransaction()'s behavior has been changed so that it will no longer return a Transaction instance for a transaction which is prepared, but not yet committed or rolled back. The only user operations which can be performed on prepared, but not yet committed/rolled-back transactions are XAEnvironment.commit() or rollback(). [#16375]

  3. Added the DatabaseConfig.setUseExistingConfig and DatabaseConfig.getUseExistingConfig methods. This allows a user application to open a Database readonly and determine its existing DatabaseConfig (e.g. whether the database has sorted duplicates or not). [#16504]

  4. Several major improvements have been made to the JE exception hierarchy. In most cases, no source code changes to applications are required. However, users are strongly encouraged to read the new exception documentation and follow the recommended approach for exception handling. In one case, a compatibility flag must be set if DeadlockException or LockNotGrantedException is explicitly caught, and the use of these exceptions is not changed as described below. Also, if an instance of DatabaseException is being created directly by the application, this must be changed as described below.

    See the DatabaseException javadoc for an overview of all JE exceptions.

  5. The Environment.getLockStats() method and LockStats class are deprecated. All of the lock stats may be obtained using Environment.getStats(). The same get methods which exist on LockStats are now present on EnvironmentStats. [#16792]

  6. Added the Database.compareKeys(DatabaseEntry, DatabaseEntry) and Database.compareDuplicates(DatabaseEntry, DatabaseEntry) methods which may be used for comparing two DatabaseEntry instances using either the btree comparator or the duplicate comparator. [#16816]

  7. The com.sleepycat.je.ForeignKeyDeleteAction, com.sleepycat.je.LockMode, and com.sleepycat.je.OperationStatus classes have been changed from classes to Java enums. This is a compatible change and does not require a recompilation unless you use LockMode.DIRTY_READ. LockMode.DIRTY_READ was previously deprecated and has now been removed from the code base. Use of the LockMode.DIRTY_READ field in class files compiled with earlier versions of JE will cause java.lang.NoSuchFieldError to be thrown at runtime.

    In addition, the following previously deprecated methods, classes, and constants have been removed:

    com.sleepycat.collections.StoredCollections.dirtyReadCollection,
    com.sleepycat.collections.StoredCollections.dirtyReadList,
    com.sleepycat.collections.StoredCollections.dirtyReadMap,
    com.sleepycat.collections.StoredCollections.dirtyReadSet,
    com.sleepycat.collections.StoredCollections.dirtyReadSortedMap,
    com.sleepycat.collections.StoredCollections.dirtyReadSortedSet,
    com.sleepycat.collections.StoredContainer.isDirtyReadAllowed,
    com.sleepycat.collections.StoredContainer.isDirtyRead,
    com.sleepycat.je.CursorConfig.DIRTY_READ,
    com.sleepycat.je.CursorConfig.setDirtyRead,
    com.sleepycat.je.CursorConfig.getDirtyRead,
    com.sleepycat.je.TransactionConfig.setDirtyRead,
    com.sleepycat.je.TransactionConfig.getDirtyRead,

    [#16984]


  8. The SecondaryCursor.dupSecondary() method has been deprecated and replaced by SecondaryCursor.dup(). The SecondaryDatabase.openSecondaryCursor() method has been deprecated and replaced by SecondaryDatabase.openCursor(). The SecondaryDatabase.getSecondaryConfig() method has been deprecated and replaced by SecondaryDatabase.getConfig(). The SecondaryDatabase.openSecondaryCursor() method has been deprecated and replaced by SecondaryDatabase.openCursor(). The cloneConfig() methods for DatabaseConfig, EvolveConfig, and StoreConfig have been deprecated and replaced by clone(). The clone() methods for CheckpointConfig, CursorConfig, EnvironmentConfig, JoinConfig, PreloadConfig, SecondaryConfig, SequenceConfig, StatsConfig, TransactionConfig, and VerifyConfig have been added. [#16985]

  9. All public void setFoo() methods in com.sleepycat.je.*Config have been changed to return this instead of void. This allows multiple set calls to be placed on one line, similar to StringBuffer calls. For example, it is now possible to do the following: 
    EnvironmentConfig envConf = new EnvironmentConfig();
envConf.setAllowCreate(true).setTransactional(true);
    This will require users to recompile their code. [#17021]

  10. In JE 4.0.70, an additional change was made so that all setter methods in the DPL StoreConfig EvolveConfig now return this rather than having a void return type. This change requires that applications using the DPL be recompiled. [#17021] (4.0.70)

  11. Previous versions of JE would allow conflicting settings in TransactionConfig and EnvironmentConfig for TxnNoSync, TxnWriteNoSync, and TxnSync. TransactionConfig no longer allows setting more than one of TxnNoSync, TxnWriteNoSync, or TxnSync to true. An IllegalArgumentException will be thrown if the settings of these configuration parameters are conflicting. [#17705]

  12. The 4.0 release provides support for replicated environments. JE defines a new class: Durability, as a single consistent way to specify durability for both standalone and replicated environments. Accordingly, the methods: setTxnNoSync, getTxnNoSync, setTxnWriteNoSync, getTxnWriteNoSync in EnvironmentMutableConfig and methods setNoSync, getNoSync, setWriteNoSync, getWriteNoSync in TransactionConfig are deprecated, the methods setDurability, getDurability should be used in their place. A new Transaction method, commit(Durability) has also been provided for use in both standalone and replicated environments.

  13. Removed any code which uses NIO for IO to the file system. Deprecated the je.log.useNIO and je.log.directNIO parameters. If EnvironmentConfig.LOG_USE_NIO or EnvironmentConfig.LOG_DIRECT_NIO is used by the application, a deprecation warning will result. If these parameters are set, they will have no effect on JE, and no warning (other than the deprecation warning) will be given to the user.

  14. JE 3.X and earlier provided various proprietary JE properties named "java.util.logging.XXX" which were meant to be set either in the je.properties file or in the EnvironmentConfig class. These properties configured JE's use of the java.util.logging framework, but did not obey the conventions of standard java.util.logging naming and configuration management. These proprietary properties have been removed. Instead, JE's use of java.util.logging can now be configured in the standard fashion through the configuration properties described by the java.util.logging package.

    To display all logging output to the console, set com.sleepycat.je.util.ConsoleHandler.level = ALL as a java.util.logging property. By default, JE records logging output of level INFO or higher is recorded in <envdir>/je.info. To change the level of output displayed in the file, set com.sleepycat.je.util.FileHandler.level to the desired level. The logging output is particularly helpful in a replicated application. An example of the output can be found here.


  15. The following Environment configuration properties now allow specifying a time unit. In the past, values could only be expressed as microseconds. For these properties, the default unit has always been microseconds and that has not changed. For more information about specifying units with configuration properties, see the Time Duration Properties section of the EnvironmentConfig class. These properties are:

    In addition, the following new methods have been added to allow specifying a unit along with a time duration. The equivalent older methods, without a unit argument, have been deprecated. The new methods are:


  16. All methods that return a new cursor in a replicated environment now allow passing a null Transaction. This includes Database.openCursor, EntityIndex.entities, EntityIndex.keys, and StoredContainer.storedIterator. Before, a null Transaction was prohibited in a replicated environment unless read-uncommitted isolation was configured. Note that a null transaction has always been allowed (and still is) in a non-replicated environment. [#16513] (4.0.70)

  17. The Transaction.commit and CurrentTransaction.commitTransaction family of methods no longer throw LockPreemptedException. Note that LockPreemptedException is only applicable in replicated environments. [#16513] (4.0.70)

  18. An IllegalArgumentException is now thrown if LockMode.RMW is passed to the Database.get and Database.getSearchBoth methods, and a null Transaction is also passed. Without a transaction it is not possible to use these methods to perform a read-modify-write operation that initially acquires a write lock, so the combination of parameters does not make sense. [#16513] (4.0.70)

  19. IllegalStateException is now thrown if Transaction.getCommitToken() is called before the transaction has finished. Previously, it would return a null commit token. [#17377] (4.0.70)

  20. Database.preload(PreloadConfig) no longer throws NullPointerException if null is passed for an argument. Instead, it uses a default PreloadConfig. [#17784] (4.0.72)

Performance and other General Changes

  1. Fixed a bug which could cause a deadlock if a Database handle is closed while another thread is still using it. Removed code which automatically closes cursors that are still open at Database.close() time. [#15413]

  2. If a call to com.sleepycat.je.Transaction.commit() throws an exception because the transaction has open cursors, a follow-on call to Transaction.abort() could fail with the following stack trace: 
        Transaction 3 has been closed. java.lang.IllegalStateException: Transaction 3 has been closed.
        at com.sleepycat.je.txn.Txn.checkState(Txn.java:1580)
        at com.sleepycat.je.txn.Txn.abortInternal(Txn.java:819)
        at com.sleepycat.je.txn.Txn.abort(Txn.java:805)
        at com.sleepycat.je.Transaction.abort(Transaction.java:90)
        at com.sleepycat.je.txn.TxnEndTest.testTxnClose(TxnEndTest.java:
    This was a bug and has been fixed so the call to abort() will complete successfully. [#16214]

  3. Improved synchronization on Database instances obviating the need to use separate Database handles in high-concurrency environments. Previously, this practice was recommended, but is no longer necessary. [#16346]

  4. Improved internal concurrency by using shared (vs exclusive) latches on internal nodes of the B+Tree. Added a new stat to Database and Environment stats to record the number of "relatches" (latch upgrades) that have occurred on both the particular Database and overall for the Environment. [#16346]

  5. When opening different environments in the same process, if recovery for one of the environments took a long time, the other environment openings would block. Changed the behavior such that opening additional environments is not blocked by another environment's recovery. [#16350]

  6. Improved JE write performance when doing synchronous transaction commits on certain Linux filesystems, including ext3.

    In JE, when the durability specified for a transaction dictates force-writing to disk (for example, a commitSync() call), it calls fsync(). On modern disk drives an fsync() will take on the order of several milliseconds. This is a relatively long time compared to the time required to do a write() call, which only moves data to the operating system's file cache.

    JE's group commit mechanism seeks to reduce the number of fsyncs required by issuing one fsync for batches of multiple transaction commits. For example, if a thread T1 requires an fsync(), and JE determines that no fsync() is in progress, it will execute that fsync() immediately. If, while T1 is executing the fsync(), some other thread(s) require an fsync(s), JE will block those threads until T1 finishes. When T1's fsync() completes, a new fsync() executes on behalf of the blocked thread(s).

    The past JE group commit implementation assumed that the underlying platform (OS + file system combination) allow IO operations like seek(), read() and write() to execute concurrently with an fsync() call (on the same file, but using different file descriptors). On Solaris and Windows this is true. Hence, on these platforms, a thread which is performing an fsync() does not block another thread performing a concurrent write(). But, on several Linux file systems, ext3 in particular, an exclusive mutex on the inode is grabbed during any IO operation. So a write() call on a file (inode) will be blocked by an fsync() operation on the same file (inode). This negates any performance improvement which might be achieved by group commit.

    The JE group commit code has been improved to batch write() and fsync() calls, rather than just fsync() calls. Just before a write() call is executed, JE checks if an fsync() is in progress and if so, the write call is queued in a (new) Write Queue. Once the fsync() completes, all pending writes in the Write Queue are executed.

    This change in behavior is enabled by default and may be disabled by setting the je.log.useWriteQueue configuration parameter to false. The size of the Write Queue (i.e. the amount of data it can queue until any currently-executing IO operations complete) can be controlled with the je.log.writeQueueSize parameter. The default for je.log.writeQueueSize is 1MB with a minimum value of 4KB and a maximum value of 32MB. The Write Queue does not use cache space controlled by the je.maxMemory parameter. [#16440]


  7. Trace messages are now written to the log at recovery and shutdown time. This applies mostly as a field support aid. [#16452]

  8. Fixed a memory leak in applications which set EnvironmentConfig.setLocking(false) and used multiple environments and shared caches. JE could continue to maintain a reference to internal data structures after an Environment is fully closed. This report was originally mentioned in this forum post. [#16453]

  9. Fixed a circular static initialization dependency between LockType and LockUpgrade. This was discovered by a user whose use of reflection caused a change in the class loading order. [#16496]

  10. Fixed a bug that causes incorrect secondary key values to be created when a partial DatabaseEntry is passed to a Cursor or Database put() method, if secondaries are configured. Before, the partial entry was passed to the SecondaryKeyCreator. Now, the resulting merged data is passed to the key creator. Thanks to archie172 for reporting this on OTN. [#16929]

  11. A bug in which the DbPrintLog utility would sometimes display negative numbers for the size of the key/data bytes when using the -S option has been fixed. A -SC option has also been added which prints the summary information in CSV format. [#17196]

  12. Fixed a bug where com.sleepycat.je.util.DbPrintLog could throw a NullPointerException. [#17456]

  13. Fixed a bug that would allow the Total Memory Usage stat to become negative on certain architectures. [#17462]

  14. Removed a concurrency hot spot in TxnManager.thread2Txn, which has a positive influence on highly concurrent applications with many transactions . [#17542]

  15. Two new environment configuration parameters are available to control the use of "proactive migration" in the log cleaner: See the javadoc for these parameters for more information. By default, proactive migration is now disabled. In earlier releases, proactive migration was unconditionally enabled for both foreground and background activities. This sometimes caused operation delays and long checkpoints, and these problems should now be reduced. But because of this change in behavior, in rare cases an application may now need to configure more cleaner threads, or enable proactive migration, in order to prevent a cleaner backlog. [#17634]

  16. In rare cases requiring a combination of delete and repeated insert operations and environment recoveries, Database.count() could be incorrect, usually by a small number of records. Note that Database.count() does have the general caveat that is it not guaranteed to be accurate when there concurrent updates. This miscount has been fixed. [#17770] (4.0.61)

  17. In JE 4.0.60, read operations such as Database.get(), when executed on a replicated environment, and used with a null transaction parameter, did not properly obey the default environment replica read consistency policy. This has been fixed. [#16513] (4.0.70)

  18. Fixed a bug that sometimes caused a NullPointerException in a replicated environment, when a database was renamed on the Master. If a secondary database was renamed, the app running on the Replica could see a NullPointerException rather than the intended DatabasePreemptedException. [#17015] (4.0.70)

  19. Fixed a bug where a LogFileNotFoundException could be seen in an environment which had repeated inserts and deletes of records with the same key and repeated environment recoveries. [#17879] (4.0.70)

  20. A bug could cause replicated environments which contain databases with duplicates to see the following exception with this distinctive stacktrace: 
    (JE 4.0.60)... LOG_FILE_NOT_FOUND: Log file missing, log is likely
 invalid. Environment is invalid and must be closed.
	at com.sleepycat.je.tree.ChildReference.
fetchTarget(ChildReference.java:147)
	at com.sleepycat.je.tree.DIN.getDupCountLN(DIN.java:155)
	at com.sleepycat.je.dbi.CursorImpl.lockDupCountLN(CursorImpl.java:2596)
	at com.sleepycat.je.tree.Tree.insertDuplicate(Tree.java:2705)
	at com.sleepycat.je.tree.Tree.insert(Tree.java:2652)
	at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:1076)
	at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:1769)
	at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:1726)
	at com.sleepycat.je.Cursor.putNotify(Cursor.java:1659)
	at com.sleepycat.je.Cursor.putLN(Cursor.java:1609)
	at com.sleepycat.je.DbInternal.putLN(DbInternal.java:125)
	at com.sleepycat.je.rep.impl.node.Replay.applyLN(Replay.java:738)
	at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:483)
	at com.sleepycat.je.rep.impl.node.Replica.doRunReplicaLoopInternalWork(Replica.java:414)
	at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:341)
	at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:283)
	at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:886)
    The problem would only occur in a fairly tight timing window, and has been fixed. [#17879] (4.0.70)

  21. A replicated environment which had been running in a replication group and restarted as a replica could see this exception at startup time: 
    
 (JE 4.0.60) ... GroupCBVLSN: 231,655 is outside VLSN range:
 first=231,700 last=583,909 sync=583,909 txnEnd=583,909
UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue.
Environment is invalid and must be closed. Problem seen replaying entry ...
  at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:392)
  at com.sleepycat.je.rep.impl.node.GlobalCBVLSN.recalculate(GlobalCBVLSN.java:179)
  at com.sleepycat.je.rep.impl.node.RepNode.recalculateGlobalCBVLSN(RepNode.java:650)
  at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:444)
  at com.sleepycat.je.rep.impl.node.Replica.doRunReplicaLoopInternalWork(Replica.java:414)
  at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:341)
  at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:283)
  at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:886)
    This has been fixed. [#17885] (4.0.70)

  22. Database.preload(PreloadConfig) no longer throws NullPointerException if null is passed for an argument. Instead, it uses a default PreloadConfig. [#17784] (4.0.72)

Direct Persistence Layer (DPL), Collections and Bind packages

  1. Report a meaningful IllegalArgumentException when @Persistent is incorrectly declared on an enum class. Before, the confusing message Persistent class has non-persistent superclass: java.lang.Enum was reported. Thanks to Lowell for reporting this problem on OTN. [#15623]

  2. Report a meaningful IllegalArgumentException when @Persistent is incorrectly declared on an interface. Before, a NullPointerException was reported. Thanks to Tony Clifton for reporting this problem on OTN. [#15841]

  3. Report a meaningful IllegalArgumentException when @Persistent or @Entity is incorrectly used on an inner class (a non-static nested class). Before, the confusing message No default constructor was reported. [#16279]

  4. Entity subclasses that define secondary keys must now be registered prior to storing an instance of the class. This can be done in two ways:

    Failure to register the entity subclass will result in an IllegalArgumentException the first time an attempt is made to store an instance of the subclass. An exception will not occur if instances of the subclass have previously been stored, which allows existing applications to run unmodified in most cases.

    This behavioral change was made to increase reliability. In several cases, registering an entity subclass has been necessary as a workaround, as described here, for example. The requirement to register the subclass will ensure that such errors do not occur in deployed applications. [#16399]


  5. When a secondary index database is opened, it is now auto-populated only if it did not previously exist, rather than if it is empty. This prevents repeated auto-population of the secondary from occurring when the secondary keys for all entities in the primary index happen to be null. This problem was reported by jhalex on OTN. Note that the workaround mentioned in the OTN thread -- to create a dummy record that contains a non-null value for each secondary key -- is no longer necessary.

    To go along with this change, the EntityStore.truncateClass method now removes all secondary databases rather than truncating them, so that they will be auto-populated if they are accessed again. The primary database is truncated, as before. [#16399]


  6. The com.sleepycat.collections.TransactionRunner.handleException method has been added to allow overriding the default transaction retry policy. See the javadoc for this method for more information. [#16574]

  7. Fixed a bug that causes an assertion to fire or a NullPointerException (when assertions are disabled) from the EntityStore constructor. The problem occurs only when the previously created EntityStore contains an entity with a secondary key definition in which the key name has been overridden and is different than the field name. The key name can be overridden with the name property of the SecondaryKey annotation. Below are examples of the assertion and exception. 
    java.lang.NullPointerException
    at com.sleepycat.persist.impl.ComplexFormat.checkSecKeyMetadata(ComplexFormat.java:1143)
    at com.sleepycat.persist.impl.ComplexFormat.evolveFieldList(ComplexFormat.java:1428)
    at com.sleepycat.persist.impl.ComplexFormat.evolveAllFields(ComplexFormat.java:1245)
    at com.sleepycat.persist.impl.ComplexFormat.evolve(ComplexFormat.java:1019)
    at com.sleepycat.persist.impl.Evolver.evolveFormatInternal(Evolver.java:440)
    at com.sleepycat.persist.impl.Evolver.evolveFormat(Evolver.java:248)
    at com.sleepycat.persist.impl.PersistCatalog.(PersistCatalog.java:357)
    at com.sleepycat.persist.impl.Store.(Store.java:180)
    at com.sleepycat.persist.EntityStore.(EntityStore.java:165)

    java.lang.AssertionError
    at com.sleepycat.persist.impl.ComplexFormat.evolveFieldList(ComplexFormat.java:1327)
    at com.sleepycat.persist.impl.ComplexFormat.evolveAllFields(ComplexFormat.java:1245)
    at com.sleepycat.persist.impl.ComplexFormat.evolve(ComplexFormat.java:1019)
    at com.sleepycat.persist.impl.Evolver.evolveFormatInternal(Evolver.java:440)
    at com.sleepycat.persist.impl.Evolver.evolveFormat(Evolver.java:248)
    at com.sleepycat.persist.impl.PersistCatalog.(PersistCatalog.java:357)
    at com.sleepycat.persist.impl.Store.(Store.java:180)
    at com.sleepycat.persist.EntityStore.(EntityStore.java:165)
    Thanks to Lukasz Antoniak for reporting this bug accurately and clearly on OTN. [#16819]

  8. Key cursors have been optimized to significantly reduce I/O when the READ_UNCOMMITTED isolation mode is used. See EntityIndex.keys for more information. [#16859]

  9. Report a meaningful IllegalArgumentException when NULLIFY is used with a @SecondaryKey and the field is a primitive type. Before, the confusing message Key field object may not be null was reported. Thanks to Erick for reporting this problem on OTN. [#17011]

  10. Enum fields may now be used as DPL keys, including primary keys, secondary keys, and fields of composite key classes. Comparators are supported for composite key classes containing enum fields.

    A bug was also fixed that prevented enum values from being inserted before existing values in an enum declaration. Now, new values may be inserted before existing values in the declaration, as well as appended to the end of the declaration. However, note that renaming and deletion of enum values is not supported. [#17140]


  11. Fixed a bug that prevented the use of custom key comparisons (composite key classes that implement Comparable) for secondary keys defined as ONE_TO_MANY or MANY_TO_MANY. An example stack trace is below. 
    java.lang.IllegalArgumentException: Key class is not a simple
  type or a composite key class (composite keys must include @KeyField
  annotations): java.util.Set
at com.sleepycat.persist.impl.PersistKeyBinding.(PersistKeyBinding.java:38)
at com.sleepycat.persist.impl.Store.getKeyBinding(Store.java:1294)
at com.sleepycat.persist.impl.Store.setBtreeComparator(Store.java:1312)
at com.sleepycat.persist.impl.Store.getSecondaryConfig(Store.java:1115)
at com.sleepycat.persist.impl.Store.openSecondaryIndex(Store.java:666)
at com.sleepycat.persist.impl.Store.openSecondaryIndexes(Store.java:636)
at com.sleepycat.persist.impl.Store.getPrimaryIndex(Store.java:384)
at com.sleepycat.persist.EntityStore.getPrimaryIndex(EntityStore.java:259)
    [#17207]

  12. Improve performance of StoredCollection.removeAll. This method no longer iterates over all records in the stored collection. Thanks to ambber on OTN for suggesting and testing this improvement. [#17727]

  13. A hot upgrade of an application (without taking down the replication group) is now allowed when persistent DPL classes have been changed. Such changes take place when the DPL schema has been updated and class evolution is required. Before, to deploy such a change, the entire replication group had to be bought down. For more information see "Upgrading a Replication Group" in the javadoc package summary for the com.sleepycat.persist.evolve package. [#16655] (4.0.70)

Utility Changes:

  1. Changed DbPrintLog -S so that it can display values in the trillions. Fixed a bug that didn't allow -S or -q to come before any other flags on the command line. [#16370]

  2. Fixed a bug which caused an InvocationTargetException to be displayed (masking the real exception) when invoking a utility with the je.jar file in this way: 
      java -jar je.jar <UtilityName>
    [#16649]

Documentation, Installation and Integration:

  1. JE support in Android has been improved by the addition of a lib/je-android.M.N.P.jar file. Rather than requiring the Android application developer to modify and recompile the JE sources so that references to javax.transaction.xa.* classes are removed, the jar file can be simply included in the project's libs directory. The JE and Android directions reflect these changes. (4.0.70)

  2. The Monitoring JE with JConsole and JMX directions had a small typo. The sentence "(e.g. using -DJEMonitor-true on the command line)" has been corrected to say "-DJEMonitor=true". (4.0.70)