Library Version 11.2.4.1, Release 4.1.10
Thread performing eviction (in this case the JEEvictor thread):
    com.sleepycat.je.recovery.DirtyINMap.coordinateEvictionWithCheckpoint(DirtyINMap.java:92)
    waiting to lock <0xaeae7118> (a com.sleepycat.je.recovery.DirtyINMap)
    at com.sleepycat.je.recovery.Checkpointer.coordinateEvictionWithCheckpoint(Checkpointer.java:203)
    at com.sleepycat.je.evictor.Evictor.coordinateWithCheckpoint(Evictor.java:1073)
    at com.sleepycat.je.evictor.Evictor.access$000(Evictor.java:124)
    at com.sleepycat.je.evictor.Evictor$1RootEvictor.doWork(Evictor.java:707)
    at com.sleepycat.je.tree.Tree.withRootLatchedExclusive(Tree.java:334)
    at com.sleepycat.je.evictor.Evictor.evictRoot(Evictor.java:741)
    at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:630)
    at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:536)
    at com.sleepycat.je.evictor.Evictor$BackgroundEvictTask.run(Evictor.java:1215)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Thread calling Environment.sync:
    sun.misc.Unsafe.park(Native Method) parking to wait for
    <0xaeafb698> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
    at com.sleepycat.je.latch.SharedLatch.acquireShared(SharedLatch.java:149)
    at com.sleepycat.je.tree.Tree.withRootLatchedShared(Tree.java:344)
    at com.sleepycat.je.dbi.DbTree.getHighestLevel(DbTree.java:1615)
    at com.sleepycat.je.recovery.DirtyINMap.selectDirtyINsForCheckpoint(DirtyINMap.java:236)
    locked <0xaeae7118> (a com.sleepycat.je.recovery.DirtyINMap)
    at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:495)
    locked <0xaeae7058> (a com.sleepycat.je.recovery.Checkpointer)
    at com.sleepycat.je.dbi.EnvironmentImpl.invokeCheckpoint(EnvironmentImpl.java:1816)
    at com.sleepycat.je.Environment.sync(Environment.java:1472) ...
[#19567]
EnvironmentConfig.setSharedCache(true) to not use the shared
cache and instead use a private cache.  This happened only when opening
multiple Environments concurrently in more than one thread.  Thanks to Tim
Underwood for
reporting
this on OTN and identifying the specific problem in the JE source code.
[#19680]
... recoveryTracker should overlap or follow on disk last VLSN of 52,678 recoveryFirst= 52,680 UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed. at com.sleepycat.je.EnvironmentFailureException.unexpectedState (EnvironmentFailureException.java:391) at com.sleepycat.je.rep.vlsn.VLSNIndex.merge(VLSNIndex.java:1230) at com.sleepycat.je.rep.vlsn.VLSNIndex.init(VLSNIndex.java:1141) at com.sleepycat.je.rep.vlsn.VLSNIndex.Thanks to Jan for reporting this problem on OTN and working with us to fix it. [#19754](VLSNIndex.java:367) at com.sleepycat.je.rep.impl.RepImpl.preRecoveryCheckpointInit(RepImpl.java:373) at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:238) at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:534) ... 
EnvironmentFailureException with
LOG_FILE_NOT_FOUND during recovery, meaning that the JE
environment cannot be opened.  We strongly recommend that all applications
using JE 4.1.6 upgrade immediately.  The bug was introduced in JE 4.1.6 and is
not present in earlier releases.  [#19346]
Exception in thread "main" java.lang.ClassCastException: com.sleepycat.persist.raw.RawObject cannot be cast to com.sleepycat.persist.model.PersistentProxy at com.sleepycat.persist.impl.ProxiedFormat.newInstance(ProxiedFormat.java:91) at com.sleepycat.persist.impl.RecordInput.readObject(RecordInput.java:174) at com.sleepycat.persist.impl.ReflectionAccessor$ObjectAccess.read(ReflectionAccessor.java:406) at com.sleepycat.persist.impl.ReflectionAccessor.readNonKeyFields(ReflectionAccessor.java:285) ...Thanks to James Li for reporting this on OTN and helping us to identify the problem in the source code. [#19312]
In this release, cache eviction is no longer serialized, and can be executed concurrently. In addition, JE now has a dedicated, configurable thread pool which will do cache eviction when memory limits are reached. Eviction is done by this dedicated pool, by other JE daemon threads, and by application threads. JE attempts to skew the eviction workload toward the pool and daemon threads, in order to offload application threads.
The eviction pool is enabled by default, but can be disabled by the je.env.runEvictor property. The properties je.evictor.coreThreads, je.evictor.maxThreads and je.evictor.keepAlive are used to configure the core, max and keepalive attribute for the pool. EnvironmentStats has new statistics that try to give an indication of whether eviction is executed by the eviction pool, by operations with specific com.sleepycat.je.CacheMode settings, by explicit calls to Environment.evictMemory, by operations that will cause the cache to go over budget, or by JE daemon threads. [#18626]
One of the optimizations reduces the in-memory footprint of an internal btree node when only a small portion of it has been referenced, as might be the case when data records are accessed in a random order, or when a subset of data is accessed. It does not help if the application is doing a database-wide cursor traversal. [#18623]
Another optimization takes effect when the key less than or equal to 16 bytes in size, which can be true either when the key is naturally small, or when key prefixing is enabled through Database.setKeyPrefix(). [#18624]
Database.sync and Database.close
for deferred-write databases, so that no file write or fsync is
performed if nothing in the database is dirty.  Calling these methods
will now have very low cost when no changes to the database have been
made. [#18402]
<DaemonThread name="INCompressor"/> caught exception: com.sleepycat.je.EnvironmentFailureException: Node 1(1):rrep0 Read invisible log entry at 0x98c/0x2c711 <hdr type="LN_TX/7" <lsn v="6,505,875"> isInvisible="1" prev="0x2c6dd" size="28" cksum="501157099"/> LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. fetchTarget of 0x98c/0x2c711 parent IN=55088 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x9c2/0x24a54 parent.getDirty()=true state=1 at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:927) at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:781) at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:742) at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1309) at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1321) at com.sleepycat.je.tree.BIN.compress(BIN.java:804) at com.sleepycat.je.incomp.INCompressor.compressBin(INCompressor.java:501) at com.sleepycat.je.incomp.INCompressor.doCompress(INCompressor.java:411) at com.sleepycat.je.incomp.INCompressor.onWakeup(INCompressor.java:341) at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:162)
(JE 4.0.92) var/db/bdb_userNode fetchTarget of 0x3151/0x44b2638 parent IN=729921304 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x3189/0x13616b0 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241) at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1300) ...Thanks to "helg" on OTN for reporting this and working with us to diagnose the problem. [#17252] (fixed in 4.0.114)
PreloadStatus.EXCEEDED_TIME to be
incorrectly returned from the Database.preload()
method. [#18577]
(fixed in 4.0.114)
     Environment env = new Environment( envHome, envConfig)
     ReplicatedEnvironment repEnv = new ReplicatedEnvironment(envHome, repConfig, envConfig)
 
she would see a java.lang.ClassCastException, instead of the more
informative java.lang.UnsupportedOperationException. This has been
fixed. Note that it is permissible to open an environment as a
ReplicatedEnvironment, and then follow by opening it as a read only
Environment. This was initially reported on
the OTN forum.  [#18649]
EnvironmentConfig.CLEANER_LAZY_MIGRATION to
provide finer control over log cleaner, checkpointing and eviction
behavior.  See the javadoc for details.  [#18650]
(fixed in 4.0.114)
Exception in thread "main" com.sleepycat.je.EnvironmentFailureException:
(JE 4.0.92) last LSN=0x424/0xe4fc1 LOG_INTEGRITY: Log information is incorrect,
problem is likely persistent. Environment is invalid and must be closed.
    at com.sleepycat.je.recovery.RecoveryManager.traceAndThrowException(RecoveryManager.java:3052)
    at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:867)
    at com.sleepycat.je.recovery.RecoveryManager.buildINs(RecoveryManager.java:621)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:513)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:175)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:529)
    at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:204)
    at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:230)
    at com.sleepycat.je.Environment.(Environment.java:212)
    at com.sleepycat.je.Environment.(Environment.java:166)
    ...
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.92)
fetchTarget of 0x89/0x4c2e29 parent IN=4883580 IN
class=com.sleepycat.je.tree.DIN lastFullVersion=0x424/0xdcec4
parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is
likely invalid. Environment is invalid and must be closed.
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
    at com.sleepycat.je.tree.DIN.fetchTarget(DIN.java:520)
    at com.sleepycat.je.tree.IN.findParent(IN.java:2704)
    at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:879)
    at com.sleepycat.je.recovery.RecoveryManager.replayINDelete(RecoveryManager.java:1770)
    at com.sleepycat.je.recovery.RecoveryManager.replayOneIN(RecoveryManager.java:945)
    at com.sleepycat.je.recovery.RecoveryManager.readINs(RecoveryManager.java:846)
    ... 11 more
Caused by: java.io.FileNotFoundException: 00000089.jdb (The system cannot find
the file specified)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Unknown Source)
    at java.io.RandomAccessFile.(Unknown Source)
    at com.sleepycat.je.log.FileManager$1.(FileManager.java:993)
    at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:992)
    at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:888)
    at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
    at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
    ... 17 more
     
The bug only occurred under the following conditions:
No data loss occurs as a result of this bug. By using a version of JE with the bug fix, such environments can be opened and used normally. Thanks to OTN user justindthomas for reporting the problem and working with us to diagnose and fix it. [#18663] (fixed in 4.0.114)
IllegalStateException when a DPL
Converter mutation is used for class evolution, and Replicas are
upgraded first (as prescribed) in a replication group. [#18690]
(fixed in 4.0.114)
Environment.removeDatabase or
Environment.truncateDatabase is called, and the program crashes or
exits before any other information is written to disk.  For example, if the
Environment is closed normally after the removal/truncation, or a scheduled
checkpoint occurs, then the bug will not occur. [#18696]
(fixed in 4.0.114)
All public JE exception and statistics classes should be serializable. Some were not, and have been fixed. The following classes are now certified to be serializable. [#18738]
com.sleepycat.je.BtreeStatscom.sleepycat.je.CommitTokencom.sleepycat.je.DatabaseEntrycom.sleepycat.je.DatabaseExceptioncom.sleepycat.je.DatabaseExistsExceptioncom.sleepycat.je.DatabaseNotFoundExceptioncom.sleepycat.je.DatabaseStatscom.sleepycat.je.DeadlockExceptioncom.sleepycat.je.DeleteConstraintExceptioncom.sleepycat.je.DuplicateDataExceptioncom.sleepycat.je.EnvironmentFailureExceptioncom.sleepycat.je.EnvironmentLockedExceptioncom.sleepycat.je.EnvironmentNotFoundExceptioncom.sleepycat.je.EnvironmentStatscom.sleepycat.je.ForeignConstraintExceptioncom.sleepycat.je.LockConflictExceptioncom.sleepycat.je.LockNotAvailableExceptioncom.sleepycat.je.LockNotGrantedExceptioncom.sleepycat.je.LockStatscom.sleepycat.je.LockTimeoutExceptioncom.sleepycat.je.LogWriteExceptioncom.sleepycat.je.OperationFailureExceptioncom.sleepycat.je.PreloadStatscom.sleepycat.je.PreloadStatuscom.sleepycat.je.RunRecoveryExceptioncom.sleepycat.je.SecondaryConstraintExceptioncom.sleepycat.je.SecondaryIntegrityExceptioncom.sleepycat.je.SecondaryReferenceExceptioncom.sleepycat.je.SequenceExistsExceptioncom.sleepycat.je.SequenceIntegrityExceptioncom.sleepycat.je.SequenceNotFoundExceptioncom.sleepycat.je.SequenceOverflowExceptioncom.sleepycat.je.SequenceStatscom.sleepycat.je.ThreadInterruptedExceptioncom.sleepycat.je.TransactionStatscom.sleepycat.je.TransactionStats.Activecom.sleepycat.je.TransactionTimeoutExceptioncom.sleepycat.je.UniqueConstraintExceptioncom.sleepycat.je.VersionMismatchExceptioncom.sleepycat.je.XAFailureExceptioncom.sleepycat.je.jca.ra.JEExceptioncom.sleepycat.je.rep.DatabasePreemptedExceptioncom.sleepycat.je.rep.GroupShutdownExceptioncom.sleepycat.je.rep.InsufficientAcksExceptioncom.sleepycat.je.rep.InsufficientLogExceptioncom.sleepycat.je.rep.InsufficientReplicasExceptioncom.sleepycat.je.rep.LockPreemptedExceptioncom.sleepycat.je.rep.LogOverwriteExceptioncom.sleepycat.je.rep.MasterStateExceptioncom.sleepycat.je.rep.MemberNotFoundExceptioncom.sleepycat.je.rep.ReplicaConsistencyExceptioncom.sleepycat.je.rep.ReplicaWriteExceptioncom.sleepycat.je.rep.ReplicatedEnvironmentStatscom.sleepycat.je.rep.RestartRequiredExceptioncom.sleepycat.je.rep.RollbackExceptioncom.sleepycat.je.rep.RollbackProhibitedExceptioncom.sleepycat.je.rep.StateChangeExceptioncom.sleepycat.je.rep.UnknownMasterExceptioncom.sleepycat.je.util.LogVerificationExceptioncom.sleepycat.persist.IndexNotAvailableExceptioncom.sleepycat.persist.StoreExistsExceptioncom.sleepycat.persist.StoreNotFoundExceptioncom.sleepycat.persist.evolve.DeletedClassExceptioncom.sleepycat.persist.evolve.IncompatibleClassExceptionNullPointerException under certain
circumstances when key prefixing is enabled. This was originally reported on the
OTN Forum. [#18773]
(fixed in 4.0.114)
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:374)
        at java.util.HashSet.add(HashSet.java:200)
        at com.sleepycat.je.dbi.EnvironmentImpl.registerExceptionListenerUser(EnvironmentImpl.java:730)
        at com.sleepycat.je.utilint.DaemonThread.(DaemonThread.java:60)
        at com.sleepycat.je.cleaner.FileProcessor.(FileProcessor.java:110)
        at com.sleepycat.je.cleaner.Cleaner.doClean(Cleaner.java:461)
        at com.sleepycat.je.dbi.EnvironmentImpl.invokeCleaner(EnvironmentImpl.java:1879)
        at com.sleepycat.je.Environment.cleanLog(Environment.java:1559)
   
java.io.FileNotFoundException: ... (The system cannot find the file specified)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Unknown Source)
    at java.io.RandomAccessFile.(Unknown Source)
    at com.sleepycat.je.log.FileManager$1.(FileManager.java:998)
    at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:998)
    at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:893)
    at com.sleepycat.je.rep.stream.FeederReader$SwitchWindow.fillNext(FeederReader.java:523)
    at com.sleepycat.je.log.FileReader.readData(FileReader.java:758)
    at com.sleepycat.je.log.FileReader.readNextEntryAllowExceptions(FileReader.java:259)
    at com.sleepycat.je.log.FileReader.readNextEntry(FileReader.java:230)
    at com.sleepycat.je.rep.stream.FeederReader.scanForwards(FeederReader.java:284)
    at com.sleepycat.je.rep.stream.MasterFeederSource.getWireRecord(MasterFeederSource.java:62)
    at com.sleepycat.je.rep.impl.node.Feeder$OutputThread.run(Feeder.java:659)
   
java.lang.ArrayIndexOutOfBoundsException at java.util.zip.Adler32.update(Adler32.java:47) at com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:59) at com.sleepycat.je.log.ChecksumValidator.update(ChecksumValidator.java:55) at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:928) ...Thanks to Jean-Christophe on OTN for reporting this bug. (fixed in 4.0.114)
java.lang.IllegalArgumentException: Can not set java.lang.Integer field com.sleepycat.persist.test.EvolveClasses$RenameSecField.new_secKey2 to java.lang.String at java.lang.reflect.Field.set(Field.java:657) at com.sleepycat.persist.impl.ReflectionAccessor$ObjectAccess.read(ReflectionAccessor.java:422) at com.sleepycat.persist.impl.ReflectionAccessor.readSecKeyFields(ReflectionAccessor.java:253) at com.sleepycat.persist.impl.ComplexFormat$EvolveReader.readObject(ComplexFormat.java:2127) at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(PersistEntityBinding.java:115) at com.sleepycat.persist.impl.PersistEntityBinding.entryToObjectInternal(PersistEntityBinding.java:83) at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(PersistEntityBinding.java:64) at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:597) at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:584) ...This has been fixed. [#18961] (fixed in 4.0.114)
Cursor.getNextDup or
EntityCursor.nextDup to advance to a following key under certain
rare conditions.  This also has a side effect of causing
Database.delete to delete duplicate records for the following key
under the same conditions.  The conditions that lead to the bug are:
MANY_TO_XXX relationship is used.Cursor.getNextDup or EntityCursor.nextDup is
    called when positioned on the last record for key A, or
    Database.delete is called for key A.  In this case, the
    record(s) with key B will mistakenly be returned or deleted.... Caused by: java.lang.IllegalAccessException: Class com.sleepycat.persist.impl.ReflectionAccessor can not access a member of class XXX with modifiers "public" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65) at java.lang.reflect.Constructor.newInstance(Constructor.java:505) at com.sleepycat.persist.impl.ReflectionAccessor.newInstance(ReflectionAccessor.java:151) ...Thanks to Sheila on OTN for reporting this problem and working with us. [#19100]
To resolve this, a new exception, com.sleepycat.je.rep.MasterReplicaTransitionException is thrown in these cases. MasterReplicaTransitionException is a RestartRequiredException, and the application must close and reopen its environment handle, thereby properly reinitializing the node. [#19177]