Oracle Berkeley DB Java Edition 11G R2 Change Log

Library Version 11.2.4.1, Release 4.1.6


Changes in 4.1.6

Performance Enhancements

When a BDB JE application has a data set that does not fit entirely in cache, AND there is no particular working set of data that fits in cache, the application will see the best performance when as much of the internal btree nodes, or index, are kept in cache. This release includes improvements to make JE's cache management more efficient, and provides statistics to help determine cache efficiency.
  1. Added concurrent eviction, which can be a benefit to any BDB JE application whose data set does not fit in cache. In past releases, cache eviction has been carried out by JE daemon threads, application threads which call JE operations, and an optional single evictor thread. The actual eviction operation was serialized, and could create a bottleneck where many threads could be seen waiting upon the method com.sleepycat.je.evictor.Evictor.doEvict().

    In this release, cache eviction is no longer serialized, and can be executed concurrently. In addition, JE now has a dedicated, configurable thread pool which will do cache eviction when memory limits are reached. Eviction is done by this dedicated pool, by other JE daemon threads, and by application threads. JE attempts to skew the eviction workload toward the pool and daemon threads, in order to offload application threads.

    The eviction pool is enabled by default, but can be disabled by the je.env.runEvictor property. The properties je.evictor.coreThreads, je.evictor.maxThreads and je.evictor.keepAlive are used to configure the core, max and keepalive attribute for the pool. EnvironmentStats has new statistics that try to give an indication of whether eviction is executed by the eviction pool, by operations with specific com.sleepycat.je.CacheMode settings, by explicit calls to Environment.evictMemory, by operations that will cause the cache to go over budget, or by JE daemon threads. [#18626]


  2. This release introduces several new kinds of in-memory btree node compression, targeted toward reducing the memory needed to hold internal btree nodes in cache. This is particularly helpful for applications which have data sets which do not fit in cache.

    One of the optimizations reduces the in-memory footprint of an internal btree node when only a small portion of it has been referenced, as might be the case when data records are accessed in a random order, or when a subset of data is accessed. It does not help if the application is doing a database-wide cursor traversal. [#18623]

    Another optimization takes effect when the key less than or equal to 16 bytes in size, which can be true either when the key is naturally small, or when key prefixing is enabled through Database.setKeyPrefix(). [#18624]


  3. Optimized Database.sync and Database.close for deferred-write databases, so that no file write or fsync is performed if nothing in the database is dirty. Calling these methods will now have very low cost when no changes to the database have been made. [#18402]

  4. The javadoc for com.sleepycat.je.Durability states that COMMIT_NO_SYNC is the default durablity for replicated transactions.The documentation is correct, but the actual default used was incorrect, and was COMMIT_SYNC. This has been fixed. [#18900]

General Changes

  1. The following exception could be seen in rare circumstances from a replicated environment. It is not an actual error, and was the result of an over zealous assertion. The assertion has been corrected.
    
    <DaemonThread name="INCompressor"/> caught exception: 
     com.sleepycat.je.EnvironmentFailureException:
     Node 1(1):rrep0 Read invisible log entry at 0x98c/0x2c711 
    <hdr type="LN_TX/7" <lsn v="6,505,875"> 
     isInvisible="1" prev="0x2c6dd" size="28" cksum="501157099"/>
     LOG_INTEGRITY:
     Log information is incorrect, problem is likely
     persistent. fetchTarget of 0x98c/0x2c711 parent IN=55088 
    IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x9c2/0x24a54 parent.getDirty()=true state=1
    
    at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:927)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:781)
    at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:742)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1309)
    at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1321)
    at com.sleepycat.je.tree.BIN.compress(BIN.java:804)
    at com.sleepycat.je.incomp.INCompressor.compressBin(INCompressor.java:501)
    at com.sleepycat.je.incomp.INCompressor.doCompress(INCompressor.java:411)
    at com.sleepycat.je.incomp.INCompressor.onWakeup(INCompressor.java:341)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:162)
    

  2. Fixed a bug that causes a EnvironmentFailureException LOG_FILE_NOT_FOUND when using a particular sequence of operations with a Database configured for duplicates (DatabaseConfig.setSortedDuplicates(true)) or a DPL secondary key with a MANY_TO_ONE or MANY_TO_MANY relationship. The sequence of operations that causes the bug is:
    1. A single record exists (no duplicates) for key A
    2. New transaction T1 starts
    3. Key A is deleted in T1
    4. Two or more duplicates are inserted for key A in T1
    5. The T1 transaction aborts
    Later, the log file containing the original version of the record is cleaned and deleted, causing an exception when that record is subsequently accessed. An example stack trace is below.
    (JE 4.0.92) var/db/bdb_userNode fetchTarget of 0x3151/0x44b2638 parent
    IN=729921304 IN class=com.sleepycat.je.tree.DBIN
    lastFullVersion=0x3189/0x13616b0 parent.getDirty()=false state=0
    LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is
    invalid and must be closed.
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
    at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1300)
    ...
    
    Thanks to "helg" on OTN for reporting this and working with us to diagnose the problem. [#17252] (fixed in 4.0.114)

  3. Added documentation for the je.rep.node.priority configuration property. [#17684] (fixed in 4.0.114)

  4. Fixed a bug which could cause PreloadStatus.EXCEEDED_TIME to be incorrectly returned from the Database.preload() method. [#18577] (fixed in 4.0.114)

  5. Improved the exception thrown in the following case: JE does not support opening an environment with first a non replicated and then a replicated environment handle. For example, if the user executes these statements in the following order:
         Environment env = new Environment( envHome, envConfig)
         ReplicatedEnvironment repEnv = new ReplicatedEnvironment(envHome, repConfig, envConfig)
     
    she would see a java.lang.ClassCastException, instead of the more informative java.lang.UnsupportedOperationException. This has been fixed. Note that it is permissible to open an environment as a ReplicatedEnvironment, and then follow by opening it as a read only Environment. This was initially reported on the OTN forum. [#18649]

  6. Added the property EnvironmentConfig.CLEANER_LAZY_MIGRATION to provide finer control over log cleaner, checkpointing and eviction behavior. See the javadoc for details. [#18650] (fixed in 4.0.114)

  7. Fixed a bug that causes a fatal exception during recovery (opening of a JE Environment) under certain conditions. An example of the exception stack trace is below.
    Exception in thread "main" com.sleepycat.je.EnvironmentFailureException:
    (JE 4.0.92) last LSN=0x424/0xe4fc1 LOG_INTEGRITY: Log information is incorrect,
    problem is likely persistent. Environment is invalid and must be closed.
        at com.sleepycat.je.recovery.RecoveryManager.traceAndThrowException(RecoveryManager.java:3052)
        at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:867)
        at com.sleepycat.je.recovery.RecoveryManager.buildINs(RecoveryManager.java:621)
        at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:513)
        at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:175)
        at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:529)
        at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:204)
        at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:230)
        at com.sleepycat.je.Environment.(Environment.java:212)
        at com.sleepycat.je.Environment.(Environment.java:166)
        ...
    Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.92)
    fetchTarget of 0x89/0x4c2e29 parent IN=4883580 IN
    class=com.sleepycat.je.tree.DIN lastFullVersion=0x424/0xdcec4
    parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is
    likely invalid. Environment is invalid and must be closed.
        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
        at com.sleepycat.je.tree.DIN.fetchTarget(DIN.java:520)
        at com.sleepycat.je.tree.IN.findParent(IN.java:2704)
        at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:879)
        at com.sleepycat.je.recovery.RecoveryManager.replayINDelete(RecoveryManager.java:1770)
        at com.sleepycat.je.recovery.RecoveryManager.replayOneIN(RecoveryManager.java:945)
        at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:846)
        ... 11 more
    Caused by: java.io.FileNotFoundException: 00000089.jdb (The system cannot find
    the file specified)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.(Unknown Source)
        at java.io.RandomAccessFile.(Unknown Source)
        at com.sleepycat.je.log.FileManager$1.(FileManager.java:993)
        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:992)
        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:888)
        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
        ... 17 more
    

    The bug only occurred under the following conditions:

    No data loss occurs as a result of this bug. By using a version of JE with the bug fix, such environments can be opened and used normally. Thanks to OTN user justindthomas for reporting the problem and working with us to diagnose and fix it. [#18663] (fixed in 4.0.114)


  8. Fixed a bug that causes an IllegalStateException when a DPL Converter mutation is used for class evolution, and Replicas are upgraded first (as prescribed) in a replication group. [#18690] (fixed in 4.0.114)

  9. Fixed a bug which can make log utilization inaccurate and prevent log cleaning, when Environment.removeDatabase or Environment.truncateDatabase is called, and the program crashes or exits before any other information is written to disk. For example, if the Environment is closed normally after the removal/truncation, or a scheduled checkpoint occurs, then the bug will not occur. [#18696] (fixed in 4.0.114)

  10. All public JE exception and statistics classes should be serializable. Some were not, and have been fixed. The following classes are now certified to be serializable. [#18738]

    (fixed in 4.0.114)

  11. Fixed a bug which caused a NullPointerException under certain circumstances when key prefixing is enabled. This was originally reported on the OTN Forum. [#18773] (fixed in 4.0.114)

  12. Fixed a bug that caused log cleaner threads to hang with the following stack trace. This could occur infrequently in applications which use EnvironmentMutableConfig.setExceptionListener() and have a high level of concurrent log cleaning. [#18791] (fixed in 4.0.114)
       java.lang.Thread.State: RUNNABLE
            at java.util.HashMap.put(HashMap.java:374)
            at java.util.HashSet.add(HashSet.java:200)
            at com.sleepycat.je.dbi.EnvironmentImpl.registerExceptionListenerUser(EnvironmentImpl.java:730)
            at com.sleepycat.je.utilint.DaemonThread.(DaemonThread.java:60)
            at com.sleepycat.je.cleaner.FileProcessor.(FileProcessor.java:110)
            at com.sleepycat.je.cleaner.Cleaner.doClean(Cleaner.java:461)
            at com.sleepycat.je.dbi.EnvironmentImpl.invokeCleaner(EnvironmentImpl.java:1879)
            at com.sleepycat.je.Environment.cleanLog(Environment.java:1559)
     

  13. Fixed a problem where Database.count() could return the wrong value for replicated databases with duplicates, after the environment had experienced a replication rollback. [#18816] (fixed in 4.0.114)

  14. Fixed a problem which was intermittent and seen only on the master node of a replication group which using synchronous transaction commits. The application would see the following stack trace. The problem was transient, no data was lost, and the system could re-open successfully. [#18882] (fixed in 4.0.114)
    java.io.FileNotFoundException: ... (The system cannot find the file specified)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.(Unknown Source)
        at java.io.RandomAccessFile.(Unknown Source)
        at com.sleepycat.je.log.FileManager$1.(FileManager.java:998)
        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:998)
        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:893)
        at com.sleepycat.je.rep.stream.FeederReader$SwitchWindow.fillNext(FeederReader.java:523)
        at com.sleepycat.je.log.FileReader.readData(FileReader.java:758)
        at com.sleepycat.je.log.FileReader.readNextEntryAllowExceptions(FileReader.java:259)
        at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:230)
        at com.sleepycat.je.rep.stream.FeederReader.scanForwards(FeederReader.java:284)
        at com.sleepycat.je.rep.stream.MasterFeederSource.getWireRecord(MasterFeederSource.java:62)
        at com.sleepycat.je.rep.impl.node.Feeder$OutputThread.run(Feeder.java:659)
    

  15. When a disk corruption occurs that overwrites a log entry with non-zero data, a checksum error is now properly reported via an EnvironmentFailureException, and tools such as DbScavenger and DbVerifyLog work properly. Earlier, if a corruption overwrote specific parts of the log entry, another exception was mistakenly throw, such as:
    java.lang.ArrayIndexOutOfBoundsException at
    java.util.zip.Adler32.update(Adler32.java:47) at
    com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:59) at
    com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:55) at
    com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:928)
    ...
    
    Thanks to Jean-Christophe on OTN for reporting this bug. (fixed in 4.0.114)

  16. Opening a com.sleepycat.bind.serial.StoredClassCatalog on a replica node in a JE HA replication group incorrectly resulted in a ReplicaWriteException. This has been fixed. Thanks to user13067358 on OTN for reporting this problem. [#18938] (fixed in 4.0.114)

  17. A DPL class evolution bug caused an exception or incorrect behavior when renaming a field, when the name change caused the alphabetical order of fields in the class to change. An example of such an exception follows. However, other types of exceptions could also occur.
    java.lang.IllegalArgumentException: Can not set java.lang.Integer field 
    com.sleepycat.persist.test.EvolveClasses$RenameSecField.new_secKey2 to java.lang.String
    at java.lang.reflect.Field.set(Field.java:657)
    at com.sleepycat.persist.impl.ReflectionAccessor$ObjectAccess.read(ReflectionAccessor.java:422)
    at com.sleepycat.persist.impl.ReflectionAccessor.readSecKeyFields(ReflectionAccessor.java:253)
    at com.sleepycat.persist.impl.ComplexFormat$EvolveReader.readObject(ComplexFormat.java:2127)
    at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(PersistEntityBinding.java:115)
    at com.sleepycat.persist.impl.PersistEntityBinding.entryToObjectInternal(PersistEntityBinding.java:83)
    at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(PersistEntityBinding.java:64)
    at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:597)
    at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:584)
    ...
    
    This has been fixed. [#18961] (fixed in 4.0.114)

  18. Fixed a bug that causes Cursor.getNextDup or EntityCursor.nextDup to advance to a following key under certain rare conditions. This also has a side effect of causing Database.delete to delete duplicate records for the following key under the same conditions. The conditions that lead to the bug are:
    1. A Database is configured for duplicates or a DPL secondary key with a MANY_TO_XXX relationship is used.
    2. The environment or transaction is explicitly configured for Serializable Isolation (this is not the default isolation mode).
    3. One or more records for key A exist.
    4. The key following key A, key B, has two or more duplicates, and the first duplicate for key B is deleted.
    5. Timing and conditions are such that the deleted duplicate for key B has not been internally compressed out of the Btree.
    6. Cursor.getNextDup or EntityCursor.nextDup is called when positioned on the last record for key A, or Database.delete is called for key A. In this case, the record(s) with key B will mistakenly be returned or deleted.
    [#19026] (fixed in 4.0.114)

  19. Fixed a DPL bug that prevented use of a non-public persistent class. Previously, an exception such as the following was thrown when the constructor was public but the class was non-public.
    ...
    Caused by: java.lang.IllegalAccessException: Class
    com.sleepycat.persist.impl.ReflectionAccessor can not access a member of class
    XXX with modifiers "public"
    at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:505)
    at com.sleepycat.persist.impl.ReflectionAccessor.newInstance(ReflectionAccessor.java:151)
    ...
    
    Thanks to Sheila on OTN for reporting this problem and working with us. [#19100]

  20. After changing stats collection settings on the JE JConsole plugin, the user may see a delay before the jconsole tab is repainted and changes take effect. This has been improved to provide a more immediate response. [#19101]

  21. Fixed a rare problem that could possibly cause a LogFileNotFoundException if the following actions occurred in a very particular ordering: [#19160]

  22. The javadoc for com.sleepycat.je.rep.ReplicatedEnvironment referred to a je.rep.group.joinGroupTimeout property. That property doesn't exist, and the documentation has been corrected to refer to je.rep.envSetupTimeout.

  23. In a JE replication system, it was possible in some cases for a master node to transfer to replica state and fail to properly reinitialize internal state. The problem is that some data record locks were not properly released, and future access to those data records could cause deadlocks.

    To resolve this, a new exception, com.sleepycat.je.rep.MasterReplicaTransitionException is thrown in these cases. MasterReplicaTransitionException is a RestartRequiredException, and the application must close and reopen its environment handle, thereby properly reinitializing the node. [#19177]