users@glassfish.java.net

Re: TopLink thread deadlock re UserTransaction.commit() in message-driven

From: <glassfish_at_javadesktop.org>
Date: Tue, 20 Nov 2007 14:29:28 PST

Hi Tom,

I set the cache transaction isolation to CONCURRENT_READ_WRITE as instructed
(see attached tar with SessionCustomizerImpl.java) and verified that the customizer class was
properly called by checking that the println() statement showed up in the log file. I also
made sure that we are using the updated patch for the toplink-essentials.jar library. As for
OneToOne relationships, we don't have any in our data model. They are all either OneToMany
(which should default to LAZY) and ManyToOne (which should default to EAGER).

However, it appears that we're back to the original problem. This time around there are
27 separate ScheduledRequestWrite message-driven bean threads that are blocked.
Interestingly, there are also two NodePollResultWrite message-driven bean threads that are
blocked as well. The jstack dump is in the attached tar file. I also included the output
from the 'imqcmd list dst' command that shows the corresponding unacknowledged messages
that match one-to-one with the blocked threads.

The time-line of the processing was as follows:
1. An application client executes and sends 353 messages to be processed by
   ScheduledRequestWrite, with 21 threads ending up blocked.
2. The same application client executes again but processing a different range of data and
   sends 543 messages to be processed by ScheduledRequestWrite, of which none get
   blocked.
3. A separate event occurs whereby two messages are sent by a different process to
   the NodePollResultTopic destination, and two NodePollResultWrite message-driven
   bean threads get blocked in a similar way. This line of processing is completely separate
   from the ScheduledRequestWrite processing and appears to occur right at the tail end
   of the application client processing from step 2.
4. A third execution of the application client (the same one from step 1 and 2) on yet a
   different data set occurs and the client sends 375 message to be processed by
   ScheduledRequestWrite, with 6 additional threads ending up blocked.

After all of this, there are 27 ScheduledRequestWrite threads blocked and 2
NodePollResultWrite threads blocked.

Some things we're wondering about:
1. Is the EntityManager.flush() call we're making during the UserTransaction as part of
   the persistAndUpdatePrimaryKey() method (as mentioned in an earlier post)
   a contributing factor to this problem? Is there a better way to obtain a database-generated
   primary key without calling EntityManager.flush() ?
2. Would the size of the TopLink cache be contributing to the problem? We have
    toplink.cache.size.default set to 1000. I know we've tried 5000 in the past, but I don't
   seem to recall this having any effect.
[Message sent by forum member 'skalynuk' (skalynuk)]

http://forums.java.net/jive/thread.jspa?messageID=246413