users@jpa-spec.java.net

[jpa-spec users] [jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...

From: Scott Marlow <smarlow_at_redhat.com>
Date: Thu, 14 Mar 2013 11:47:36 -0400

On 03/13/2013 09:42 PM, Ireland, Evan wrote:
> Responsibility should be on the agent that calls EntityManager.close.
>
> If the TX timed out, presumably that is the container.
>
> If the DBMS forces rollback (e.g. deadlock), then presumably the persistence provider would detect that case.

When the DBMS forces rollback, there is no concurrency problem today
that I know of. I'm not sure how clear I was before, probably not
enough. The concurrency issue is about a background thread calling
EntityManager.close() while the application thread is in the middle of
another EntityManager.close().

We all know that EnitityManager is not required to be thread safe,
although some implementations offer thread safety as an option at the
persistence unit definition level (which could slow application
performance if enabled from the javadoc comments :).

My "stop the presses" cry about this situation, is to bring awareness
about this potential problem. I didn't know if others were aware or
not. Now that we are all more aware of the issue, I think (everyone's)
customers will expect us to fix the concurrency issue in our
implementations whether we come to an agreement or not, to how we can
avoid concurrency errors.

Having said that, I'm very interested in hearing about the pain that
your users are seeing and how we might be able to help when the DBMS
returns a "deadlock detected error".

If I understand your proposal, the fix would be to throw a specific
error instead of the IllegalStateException. Do you have a link pointing
to a case that is accessible to us, that shows an example of an
application getting a "deadlock detected" error and the exception call
stack that goes with it?

Scott

>
> -----Original Message-----
> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
> Sent: Thursday, 14 March 2013 2:39 p.m.
> To: jsr338-experts_at_jpa-spec.java.net
> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>
>
>
> On 3/13/2013 6:35 PM, Ireland, Evan wrote:
>> Can we require that if the reason that EntityManager.close was called is due to some background (or database server) activity, then PersistenceException rather than IllegalStateException should be thrown by the EntityManager?
>>
>
> At the point at which close is called, or other? I.e., by the container or the persistence provider?
>
>> -----Original Message-----
>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>> Sent: Thursday, 14 March 2013 2:33 p.m.
>> To: jsr338-experts_at_jpa-spec.java.net
>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>
>>
>>
>> On 3/13/2013 6:26 PM, Ireland, Evan wrote:
>>> Linda,
>>>
>>> If containers were that easy to build we wouldn't have jobs.
>>>
>>> We need to think of the developer who is using the EntityManager!
>>>
>>
>> Yes. Interestingly enough, however, this particularly issue hasn't been raised
>> here in the 7 years since the release of JPA 1.0 :-)
>>
>> Given that the Java EE 7 release is set to close very shortly, I don't think we
>> can realistically impose this requirement on containers at this point.
>>
>>
>>> -----Original Message-----
>>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>>> Sent: Thursday, 14 March 2013 2:22 p.m.
>>> To: jsr338-experts_at_jpa-spec.java.net
>>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>
>>>
>>>
>>> On 3/13/2013 6:16 PM, Ireland, Evan wrote:
>>>> Linda,
>>>>
>>>> It would seem reasonable for EntityManager.close to be called at whatever time it would have been called if the transaction had not timed out (or deadlocked, etc).
>>>>
>>>
>>> Well, FWIW that was my initial thought. However, this imposes more complexity on the container implementation, so
>>> I'm not sure that it should be made a requirement.
>>>
>>>
>>>> -----Original Message-----
>>>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>>>> Sent: Thursday, 14 March 2013 1:57 p.m.
>>>> To: jsr338-experts_at_jpa-spec.java.net; Ireland, Evan
>>>> Subject: Re: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>>
>>>> Hi Evan,
>>>>
>>>> On 3/13/2013 2:14 PM, Ireland, Evan wrote:
>>>>> Folks,
>>>>>
>>>>> I do have a concern about the case where EntityManager.close is called by a background thread (e.g. transaction timeout). And who knows when I should have brought this up, but I just thought of it now and it reminded me of some old related issues from entity beans with CMP.
>>>>>
>>>>> My concern is that the application thread that is using the EntityManager may start receiving IllegalStateExceptions. Conceivably everywhere they use an EntityManager they would have to add try/catch blocks (or alter existing ones, if they currently code only for PersistenceException).
>>>>>
>>>>> IllegalStateException should generally be thrown only due to an API usage error by the "main" application thread, i.e. the one that "owns" the EntityManager, not because of the action of a background thread. It makes the API very fragile if IllegalStateException can just start cropping up any old time, at the whim of the container.
>>>>>
>>>>> Much preferable would be a subclass of PersistenceException that is thrown when an EntityManager call cannot be completed because the transaction has been marked for rollback (or already rolled back by either the DBMS or by the container), but any "close" call on the EntityManager should be application-issued, not issued by a background thread.
>>>>>
>>>>> For a bit of background, we encountered a somewhat similar and very difficult case a few years back that never really had a wonderful solution, except for "marking the container's transaction for rollback" in the event of an unexpected SQLException (e.g. other than for duplicate key). In that case, it was observed when using entity beans with CMP. Let me explain...
>>>>>
>>>>> Suppose you have a container-managed transaction, and the container uses explicit begin transaction / commit transaction pairs to delimit transactions. Now with some databases, in the case of deadlock, the DBMS may decide to rollback the transaction immediately, without waiting for the container to issue a commit or rollback statement. Now the JDBC connection is in a very interesting state: depending on the DBMS, it might have started a new transaction, or it might now be in auto-commit mode. Whereas the container thinks the DBMS transaction is still active, although it might have noticed that the deadlocked statement threw a SQLException. Depending on the container implementation, this problem can be compounded by session beans that catch exceptions and "eat" them (i.e. not rethrowing or wrapping them).
>>>>>
>>>>> Now back to JPA, my take on this is:
>>>>>
>>>>> If the DBMS unilaterally rolls back a transaction, or the container does, then mark the container's transaction for rollback, and ensure that subsequent EntityManager calls that cannot be sensibly (or safely) completed due to the rollback that already happened, will result in a PersistenceException (or some new subclass of it). Critically, leave the EntityManager itself alone, don't call 'close' on it, and don't throw IllegalStateException unexpectedly.
>>>>>
>>>>
>>>> I understand your point about throwing the PersistenceException. However, when do you expect the container
>>>> to call EntityManager.close()?
>>>>
>>>>> -----Original Message-----
>>>>> From: Scott Marlow [mailto:smarlow_at_redhat.com]
>>>>> Sent: Thursday, 14 March 2013 8:10 a.m.
>>>>> To: jsr338-experts_at_jpa-spec.java.net
>>>>> Cc: Linda DeMichiel
>>>>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>>>
>>>>> On 03/13/2013 02:15 PM, Linda DeMichiel wrote:
>>>>>> Hi Scott,
>>>>>>
>>>>>> On 3/13/2013 11:04 AM, Scott Marlow wrote:
>>>>>>> Are others responding privately perhaps? :)
>>>>>>>
>>>>>>
>>>>>> No. I wish they were responding in *any* manner!
>>>>>>
>>>>>>
>>>>>>> At a minimum, I would like to state that the JTA transaction could be
>>>>>>> rolled back from an external thread in the
>>>>>>> following sections:
>>>>>>>
>>>>>>
>>>>>> As you point out, the JTA spec already allows this, so if that is all we
>>>>>> were to do, I'm not sure I
>>>>>> see the point. In case I am being dense though, can you tell me what
>>>>>> words you would like to see added to
>>>>>> the spec.
>>>>>
>>>>> For 7.9.1, how about something like:
>>>>>
>>>>> "
>>>>> After the JTA transaction has completed (either by transaction commit
>>>>> or rollback), the container closes the entity manager by calling
>>>>> EntityManager.close. The JTA transaction may rollback in a background
>>>>> thread (e.g. transaction timeout), in which case, the container should
>>>>> arrange for the entity manager to be closed but the
>>>>> EntityManager.close() should not be concurrently called while the
>>>>> application is in an EntityManager invocation.
>>>>> "
>>>>>
>>>>> The 7.9.2 wording can be similar I think:
>>>>> "
>>>>> When the JTA transaction rolls back, the provider must detach all
>>>>> managed entities if the persistence context is of type
>>>>> SynchronizationType.SYNCHRONIZED or has otherwise been joined to the
>>>>> transaction. The JTA transaction may rollback in a background thread
>>>>> (e.g. transaction timeout), in which case, the provider should arrange
>>>>> for the managed entities to be detached from the persistence context but
>>>>> not concurrently while the application is in an EntityManager invocation.
>>>>> "
>>>>>
>>>>>>
>>>>>> To me, the real issue seems to be whether we can/should provide any
>>>>>> guidance as to how to handle such
>>>>>> situations. I'd like to get the benefit of hearing from the vendors
>>>>>> here as to what their implementations
>>>>>> do.
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> -Linda
>>>>>>
>>>>>>
>>>>>>> The current wording is:
>>>>>>>
>>>>>>> "
>>>>>>> 7.9.1 Container Responsibilities
>>>>>>> ...
>>>>>>> * After the JTA transaction has completed (either by transaction
>>>>>>> commit or rollback), the container closes the entity
>>>>>>> manager calling EntityManager.close.
>>>>>>> ...
>>>>>>>
>>>>>>> 7.9.2 Provider Responsibilities
>>>>>>> ...
>>>>>>> * When the JTA transaction rolls back, the provider must detach all
>>>>>>> managed entities if the persistence context is of
>>>>>>> type SynchronizationType.SYNCHRONIZED or has otherwise been joined to
>>>>>>> the transaction.
>>>>>>> ...
>>>>>>> "
>>>>>>>
>>>>>>>
>>>>>>> On 03/08/2013 08:31 PM, Linda DeMichiel wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/7/2013 4:48 PM, Scott Marlow wrote:
>>>>>>>>> On 03/07/2013 05:41 PM, Linda DeMichiel wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/6/2013 2:16 PM, Scott Marlow wrote:
>>>>>>>>>>> [1] requires that at Transaction completion, the container closes (or
>>>>>>>>>>> returns to cache), the transaction-scoped
>>>>>>>>>>> persistence context. What is supposed to happen when the JTA
>>>>>>>>>>> transaction completes in a different thread than the
>>>>>>>>>>> application thread? For example, if a background thread calls the
>>>>>>>>>>> Synchronization.afterCompletion() because the tx
>>>>>>>>>>> timeout period has been exceeded (as some Transaction Managers may
>>>>>>>>>>> do), its not exactly thread-safe to call
>>>>>>>>>>> EntityManager.close() (see [2]). Specifically, the application could
>>>>>>>>>>> be in the middle of a persist or some other
>>>>>>>>>>> EntityManager method, when EntityManager.close() is called.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The team here tells me that this should not be happening, and that the
>>>>>>>>>> transaction managers they are
>>>>>>>>>> familiar with will just mark the transaction for rollback rather than
>>>>>>>>>> rolling it back at the point
>>>>>>>>>> of timeout.
>>>>>>>>>
>>>>>>>>> Currently, we roll the transaction back from the background timer
>>>>>>>>> thread. The JTA spec [4] does allow different threads
>>>>>>>>> to start/end the transaction.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, I realize this is permitted.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nevertheless, if the container were working with a TM where a timeout
>>>>>>>>>> did result in immediate
>>>>>>>>>> rollback and invocation of afterCompletion, the container should note
>>>>>>>>>> this, and at the point at
>>>>>>>>>> which the transaction would normally be completed then do the actual
>>>>>>>>>> close as it normally would.
>>>>>>>>>
>>>>>>>>> Should we include a form of the above text in the JPA 2.1 spec
>>>>>>>>> (section 7.9.1 [1])?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Unfortunately, I don't think this may always work, because the
>>>>>>>> container may be relying on synchronization notifications at the
>>>>>>>> normally expected tx end to know when it should be calling close
>>>>>>>> (i.e., it may not know when the tx was started). If EJB CMT were
>>>>>>>> used, the container would know when a tx was started and could use a
>>>>>>>> business method boundary as the interpositioning point. If a
>>>>>>>> container wrapped UserTransaction, I suppose it could use that point
>>>>>>>> as well, but it is not obvious to me how this would be handled
>>>>>>>> otherwise.
>>>>>>>>
>>>>>>>> How does your implementation handle this?
>>>>>>>>
>>>>>>>> I'd also like to hear from the other implementations here as to
>>>>>>>> what they do and how their transaction manager implementations
>>>>>>>> handle timeout.
>>>>>>>>
>>>>>>>>
>>>>>>>>> How would we word what the provider side has to do when detaching
>>>>>>>>> entities after rollback [3]? I'm not sure that the
>>>>>>>>> persistence provider will have the same chance to make a note for the
>>>>>>>>> container side to take action on (if there is an
>>>>>>>>> EE container involved). There is also the expectation that any JPA
>>>>>>>>> provider will work, with any EE container to consider.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What do your transaction manager and container do?
>>>>>>>>>>
>>>>>>>>>>> Related to the above, if a JTA transaction rollback occurs in a
>>>>>>>>>>> background thread [3], how are the managed entities
>>>>>>>>>>> expected to be detached without violating the EntityManager
>>>>>>>>>>> thread-safety [2]?
>>>>>>>>>>>
>>>>>>>>>>> There may be vendor specific solutions but shouldn't we (JPA spec eg)
>>>>>>>>>>> account for the interaction of thread-unsafe
>>>>>>>>>>> persistence contexts and the JTA Synchronization.afterCompletion that
>>>>>>>>>>> may be invoked in non-application (background)
>>>>>>>>>>> threads?
>>>>>>>>>>>
>>>>>>>>>>> Scott
>>>>>>>>>>>
>>>>>>>>>>> [1] 7.9.1 Container Responsibilities - After the JTA transaction has
>>>>>>>>>>> completed (either by transaction commit or
>>>>>>>>>>> rollback), the container closes the entity manager calling
>>>>>>>>>>> EntityManager.close.
>>>>>>>>>>>
>>>>>>>>>>> [2] 7.2 Obtaining an EntityManager - An entity manager must not be
>>>>>>>>>>> shared among multiple concurrently executing threads,
>>>>>>>>>>> as the entity manager and persistence context are not required to be
>>>>>>>>>>> threadsafe. Entity managers must only be accessed
>>>>>>>>>>> in a single-threaded manner.
>>>>>>>>>>>
>>>>>>>>>>> [3] 7.9.2 Provider Responsibilities - When the JTA transaction rolls
>>>>>>>>>>> back, the provider must detach all managed entities
>>>>>>>>>>> if the persistence context is of type
>>>>>>>>>>> SynchronizationType.SYNCHRONIZED
>>>>>>>>>>> or has otherwise been joined to the transaction.
>>>>>>>>>
>>>>>>>>> [4] JTA 1.1 spec 3.4.3 Thread of Control:
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> The X/Open XA interface specifies that the transaction association
>>>>>>>>> related xa calls must be invoked from the same thread
>>>>>>>>> context. This thread-of-control requirement is not applicable to the
>>>>>>>>> object-oriented component-based application
>>>>>>>>> run-time environment, in which application threads are dispatched
>>>>>>>>> dynamically at method invocation time. Different Java
>>>>>>>>> threads may be using the same connection resource to access the
>>>>>>>>> resource manager if the connection spans multiple method
>>>>>>>>> invocation. Depending on the implementation of the application server,
>>>>>>>>> different Java threads may be involved with the
>>>>>>>>> same XAResource object. The resource context and the transaction
>>>>>>>>> context may be operated independent of thread context.
>>>>>>>>> This means, for example, that it's possible for different threads to
>>>>>>>>> be invoking the XAResource.start and XAResource.end
>>>>>>>>> methods.
>>>>>>>>>
>>>>>>>>> If the application server allows multiple threads to use a single
>>>>>>>>> XAResource object and the associated connection to the
>>>>>>>>> resource manager, it is the responsibility of the application server
>>>>>>>>> to ensure that there is only one transaction
>>>>>>>>> context associated with the resource at any point of time.
>>>>>>>>>
>>>>>>>>> Thus the XAResource interface specified in this document requires that
>>>>>>>>> the resource managers be able to support the
>>>>>>>>> two-phase commit protocol from any thread context.
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>
>>>>>
>>>>>
>>>
>>
>