Evan,
From my point of view, there are multiple aspects to this discussion.
Some of the aspects could be handled by non-JPA technologies that may
not exist yet. I think that the problem of avoiding concurrent
EntityManager invocations has been around for a long time but just
didn't become a priority until now.
To meet the new JPA 2.1 requirements, the thread-unsafe EntityManager
needs to be protected from concurrent access, however that can be
accomplished by the container/provider respectfully.
I think that some of the related aspects are:
1. Informing the application that the tx timed out so that the *state
of the world* may be collected (this is more of a management aspect than
JPA IMO). Currently, this is already possible (in vendor specific ways)
and I'm not sure if we would want to standardize the notification of
this event occurring somewhere (in some EE.future version). In a
previous (pre EE) Java application server that I'm familiar with, I
liked how application code could receive notifications of events like
this (e.g. applications could be notified if a database connection went
bad so that the application could help notify someone to restart the db
server).
2. Having a consistent way for the application thread to learn that a
tx timeout occurred. I think that this was the point of your use case
below but I don't believe the application will always see an
IllegalStateException, although it could see an ISE. The application
will eventually see that the transaction ended but that might not happen
until the CMT method returns (or BMT logic tries to end the TX that no
longer exists). Previous to the new 2.1 requirement for handling
concurrency, applications could of seen various IllegalState like
exceptions (e.g. NullPointerException). Can we depend on the
application *eventually* seeing that the tx disappeared when the
application tries to end the TX? Do we need more than that (in some
EE.future version?)
Scott
On 03/14/2013 04:09 PM, Ireland, Evan wrote:
> Scott,
>
> I did not mean to suggest that DBMS-issued rollback would cause a concurrency problem with the EntityManager. But I shared that as an example of something that might affect the state of the current transaction unbeknownst to the application thread. We saw it with CMP entity beans when working with the SPECjAppServer2004 performance benchmark a few years back. I don't have any current test cases to share.
>
> If we come back to JPA, I am concerned not of a concurrency issue that two threads are using the EntityManager at the same time, but a scenario which I try to emphasize with examples (T1 is "application" thread, T2 is "background" thread).
>
> T1: EM.doSomething(); // suceeeds
> T1: EM.close(); // succeeds
> T1: EM.doSomething(); // throws IllegalStateException, because application thread has violated the contract of using the EM after close
>
> Now we fix the app code:
>
> T1: EM.doSomething(); // suceeeds
> T1: EM.doSomething(); // succeeds
> T1: EM.close(); // succeeds
>
> All good. Many (most) applications will not be coded to catch IllegalStateException on EM calls as those applications are not (particularly) buggy, and therefore don't need to cope with handling with their own bugs :-)
>
> Now one day, due to TX timeout, we see a new pattern:
>
> T1: EM.doSomething(); // suceeeds
> T2: EM.close(); // succeeds (note this is done by background thread)
> T1: EM.doSomething(); // throws IllegalStateException, because background thread "unexpectedly" messed with the EM
>
> What I am saying is that if we allow T2 to 'close' the EM while T1 may be using it (not necessarily "concurrently", but even in a sequential manner), then the final call should throw a PersistenceException ("TX timed out"), not an IllegalStateException ("Bad API usage"). Thread T1 has not "done anything wrong", it is just unlucky, and unluckiness deserves to be distinguished. Another way to look at this is that the fact that the TX timeout just introduced another reason for PersistenceExceptions to be thrown (not much different from when an engineer trips over the power cord for the DBMS server, and the persistence provider cannot talk to it - that shouldn't result in IllegalStateException either).
>
> Perhaps we need a different kind of "close" (e.g. "abort") on the EM that the background thread can call, in order to get this right. EM.isOpen() should still return true, but perhaps EM.isDead() or somesuch could return true.
>
> -----Original Message-----
> From: Scott Marlow [mailto:smarlow_at_redhat.com]
> Sent: Friday, 15 March 2013 4:48 a.m.
> To: jsr338-experts_at_jpa-spec.java.net
> Cc: Ireland, Evan
> Subject: Re: new sub-thread about handing "deadlock detected" error from the database...
>
>
> On 03/13/2013 09:42 PM, Ireland, Evan wrote:
>> Responsibility should be on the agent that calls EntityManager.close.
>>
>> If the TX timed out, presumably that is the container.
>>
>> If the DBMS forces rollback (e.g. deadlock), then presumably the persistence provider would detect that case.
>
> When the DBMS forces rollback, there is no concurrency problem today
> that I know of. I'm not sure how clear I was before, probably not
> enough. The concurrency issue is about a background thread calling
> EntityManager.close() while the application thread is in the middle of
> another EntityManager.close().
>
> We all know that EnitityManager is not required to be thread safe,
> although some implementations offer thread safety as an option at the
> persistence unit definition level (which could slow application
> performance if enabled from the javadoc comments :).
>
> My "stop the presses" cry about this situation, is to bring awareness
> about this potential problem. I didn't know if others were aware or
> not. Now that we are all more aware of the issue, I think (everyone's)
> customers will expect us to fix the concurrency issue in our
> implementations whether we come to an agreement or not, to how we can
> avoid concurrency errors.
>
> Having said that, I'm very interested in hearing about the pain that
> your users are seeing and how we might be able to help when the DBMS
> returns a "deadlock detected error".
>
> If I understand your proposal, the fix would be to throw a specific
> error instead of the IllegalStateException. Do you have a link pointing
> to a case that is accessible to us, that shows an example of an
> application getting a "deadlock detected" error and the exception call
> stack that goes with it?
>
> Scott
>
>>
>> -----Original Message-----
>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>> Sent: Thursday, 14 March 2013 2:39 p.m.
>> To: jsr338-experts_at_jpa-spec.java.net
>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>
>>
>>
>> On 3/13/2013 6:35 PM, Ireland, Evan wrote:
>>> Can we require that if the reason that EntityManager.close was called is due to some background (or database server) activity, then PersistenceException rather than IllegalStateException should be thrown by the EntityManager?
>>>
>>
>> At the point at which close is called, or other? I.e., by the container or the persistence provider?
>>
>>> -----Original Message-----
>>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>>> Sent: Thursday, 14 March 2013 2:33 p.m.
>>> To: jsr338-experts_at_jpa-spec.java.net
>>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>
>>>
>>>
>>> On 3/13/2013 6:26 PM, Ireland, Evan wrote:
>>>> Linda,
>>>>
>>>> If containers were that easy to build we wouldn't have jobs.
>>>>
>>>> We need to think of the developer who is using the EntityManager!
>>>>
>>>
>>> Yes. Interestingly enough, however, this particularly issue hasn't been raised
>>> here in the 7 years since the release of JPA 1.0 :-)
>>>
>>> Given that the Java EE 7 release is set to close very shortly, I don't think we
>>> can realistically impose this requirement on containers at this point.
>>>
>>>
>>>> -----Original Message-----
>>>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>>>> Sent: Thursday, 14 March 2013 2:22 p.m.
>>>> To: jsr338-experts_at_jpa-spec.java.net
>>>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>>
>>>>
>>>>
>>>> On 3/13/2013 6:16 PM, Ireland, Evan wrote:
>>>>> Linda,
>>>>>
>>>>> It would seem reasonable for EntityManager.close to be called at whatever time it would have been called if the transaction had not timed out (or deadlocked, etc).
>>>>>
>>>>
>>>> Well, FWIW that was my initial thought. However, this imposes more complexity on the container implementation, so
>>>> I'm not sure that it should be made a requirement.
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Linda DeMichiel [mailto:linda.demichiel_at_oracle.com]
>>>>> Sent: Thursday, 14 March 2013 1:57 p.m.
>>>>> To: jsr338-experts_at_jpa-spec.java.net; Ireland, Evan
>>>>> Subject: Re: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>>>
>>>>> Hi Evan,
>>>>>
>>>>> On 3/13/2013 2:14 PM, Ireland, Evan wrote:
>>>>>> Folks,
>>>>>>
>>>>>> I do have a concern about the case where EntityManager.close is called by a background thread (e.g. transaction timeout). And who knows when I should have brought this up, but I just thought of it now and it reminded me of some old related issues from entity beans with CMP.
>>>>>>
>>>>>> My concern is that the application thread that is using the EntityManager may start receiving IllegalStateExceptions. Conceivably everywhere they use an EntityManager they would have to add try/catch blocks (or alter existing ones, if they currently code only for PersistenceException).
>>>>>>
>>>>>> IllegalStateException should generally be thrown only due to an API usage error by the "main" application thread, i.e. the one that "owns" the EntityManager, not because of the action of a background thread. It makes the API very fragile if IllegalStateException can just start cropping up any old time, at the whim of the container.
>>>>>>
>>>>>> Much preferable would be a subclass of PersistenceException that is thrown when an EntityManager call cannot be completed because the transaction has been marked for rollback (or already rolled back by either the DBMS or by the container), but any "close" call on the EntityManager should be application-issued, not issued by a background thread.
>>>>>>
>>>>>> For a bit of background, we encountered a somewhat similar and very difficult case a few years back that never really had a wonderful solution, except for "marking the container's transaction for rollback" in the event of an unexpected SQLException (e.g. other than for duplicate key). In that case, it was observed when using entity beans with CMP. Let me explain...
>>>>>>
>>>>>> Suppose you have a container-managed transaction, and the container uses explicit begin transaction / commit transaction pairs to delimit transactions. Now with some databases, in the case of deadlock, the DBMS may decide to rollback the transaction immediately, without waiting for the container to issue a commit or rollback statement. Now the JDBC connection is in a very interesting state: depending on the DBMS, it might have started a new transaction, or it might now be in auto-commit mode. Whereas the container thinks the DBMS transaction is still active, although it might have noticed that the deadlocked statement threw a SQLException. Depending on the container implementation, this problem can be compounded by session beans that catch exceptions and "eat" them (i.e. not rethrowing or wrapping them).
>>>>>>
>>>>>> Now back to JPA, my take on this is:
>>>>>>
>>>>>> If the DBMS unilaterally rolls back a transaction, or the container does, then mark the container's transaction for rollback, and ensure that subsequent EntityManager calls that cannot be sensibly (or safely) completed due to the rollback that already happened, will result in a PersistenceException (or some new subclass of it). Critically, leave the EntityManager itself alone, don't call 'close' on it, and don't throw IllegalStateException unexpectedly.
>>>>>>
>>>>>
>>>>> I understand your point about throwing the PersistenceException. However, when do you expect the container
>>>>> to call EntityManager.close()?
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Scott Marlow [mailto:smarlow_at_redhat.com]
>>>>>> Sent: Thursday, 14 March 2013 8:10 a.m.
>>>>>> To: jsr338-experts_at_jpa-spec.java.net
>>>>>> Cc: Linda DeMichiel
>>>>>> Subject: [jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...
>>>>>>
>>>>>> On 03/13/2013 02:15 PM, Linda DeMichiel wrote:
>>>>>>> Hi Scott,
>>>>>>>
>>>>>>> On 3/13/2013 11:04 AM, Scott Marlow wrote:
>>>>>>>> Are others responding privately perhaps? :)
>>>>>>>>
>>>>>>>
>>>>>>> No. I wish they were responding in *any* manner!
>>>>>>>
>>>>>>>
>>>>>>>> At a minimum, I would like to state that the JTA transaction could be
>>>>>>>> rolled back from an external thread in the
>>>>>>>> following sections:
>>>>>>>>
>>>>>>>
>>>>>>> As you point out, the JTA spec already allows this, so if that is all we
>>>>>>> were to do, I'm not sure I
>>>>>>> see the point. In case I am being dense though, can you tell me what
>>>>>>> words you would like to see added to
>>>>>>> the spec.
>>>>>>
>>>>>> For 7.9.1, how about something like:
>>>>>>
>>>>>> "
>>>>>> After the JTA transaction has completed (either by transaction commit
>>>>>> or rollback), the container closes the entity manager by calling
>>>>>> EntityManager.close. The JTA transaction may rollback in a background
>>>>>> thread (e.g. transaction timeout), in which case, the container should
>>>>>> arrange for the entity manager to be closed but the
>>>>>> EntityManager.close() should not be concurrently called while the
>>>>>> application is in an EntityManager invocation.
>>>>>> "
>>>>>>
>>>>>> The 7.9.2 wording can be similar I think:
>>>>>> "
>>>>>> When the JTA transaction rolls back, the provider must detach all
>>>>>> managed entities if the persistence context is of type
>>>>>> SynchronizationType.SYNCHRONIZED or has otherwise been joined to the
>>>>>> transaction. The JTA transaction may rollback in a background thread
>>>>>> (e.g. transaction timeout), in which case, the provider should arrange
>>>>>> for the managed entities to be detached from the persistence context but
>>>>>> not concurrently while the application is in an EntityManager invocation.
>>>>>> "
>>>>>>
>>>>>>>
>>>>>>> To me, the real issue seems to be whether we can/should provide any
>>>>>>> guidance as to how to handle such
>>>>>>> situations. I'd like to get the benefit of hearing from the vendors
>>>>>>> here as to what their implementations
>>>>>>> do.
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> -Linda
>>>>>>>
>>>>>>>
>>>>>>>> The current wording is:
>>>>>>>>
>>>>>>>> "
>>>>>>>> 7.9.1 Container Responsibilities
>>>>>>>> ...
>>>>>>>> * After the JTA transaction has completed (either by transaction
>>>>>>>> commit or rollback), the container closes the entity
>>>>>>>> manager calling EntityManager.close.
>>>>>>>> ...
>>>>>>>>
>>>>>>>> 7.9.2 Provider Responsibilities
>>>>>>>> ...
>>>>>>>> * When the JTA transaction rolls back, the provider must detach all
>>>>>>>> managed entities if the persistence context is of
>>>>>>>> type SynchronizationType.SYNCHRONIZED or has otherwise been joined to
>>>>>>>> the transaction.
>>>>>>>> ...
>>>>>>>> "
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/08/2013 08:31 PM, Linda DeMichiel wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/7/2013 4:48 PM, Scott Marlow wrote:
>>>>>>>>>> On 03/07/2013 05:41 PM, Linda DeMichiel wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 3/6/2013 2:16 PM, Scott Marlow wrote:
>>>>>>>>>>>> [1] requires that at Transaction completion, the container closes (or
>>>>>>>>>>>> returns to cache), the transaction-scoped
>>>>>>>>>>>> persistence context. What is supposed to happen when the JTA
>>>>>>>>>>>> transaction completes in a different thread than the
>>>>>>>>>>>> application thread? For example, if a background thread calls the
>>>>>>>>>>>> Synchronization.afterCompletion() because the tx
>>>>>>>>>>>> timeout period has been exceeded (as some Transaction Managers may
>>>>>>>>>>>> do), its not exactly thread-safe to call
>>>>>>>>>>>> EntityManager.close() (see [2]). Specifically, the application could
>>>>>>>>>>>> be in the middle of a persist or some other
>>>>>>>>>>>> EntityManager method, when EntityManager.close() is called.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The team here tells me that this should not be happening, and that the
>>>>>>>>>>> transaction managers they are
>>>>>>>>>>> familiar with will just mark the transaction for rollback rather than
>>>>>>>>>>> rolling it back at the point
>>>>>>>>>>> of timeout.
>>>>>>>>>>
>>>>>>>>>> Currently, we roll the transaction back from the background timer
>>>>>>>>>> thread. The JTA spec [4] does allow different threads
>>>>>>>>>> to start/end the transaction.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, I realize this is permitted.
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nevertheless, if the container were working with a TM where a timeout
>>>>>>>>>>> did result in immediate
>>>>>>>>>>> rollback and invocation of afterCompletion, the container should note
>>>>>>>>>>> this, and at the point at
>>>>>>>>>>> which the transaction would normally be completed then do the actual
>>>>>>>>>>> close as it normally would.
>>>>>>>>>>
>>>>>>>>>> Should we include a form of the above text in the JPA 2.1 spec
>>>>>>>>>> (section 7.9.1 [1])?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't think this may always work, because the
>>>>>>>>> container may be relying on synchronization notifications at the
>>>>>>>>> normally expected tx end to know when it should be calling close
>>>>>>>>> (i.e., it may not know when the tx was started). If EJB CMT were
>>>>>>>>> used, the container would know when a tx was started and could use a
>>>>>>>>> business method boundary as the interpositioning point. If a
>>>>>>>>> container wrapped UserTransaction, I suppose it could use that point
>>>>>>>>> as well, but it is not obvious to me how this would be handled
>>>>>>>>> otherwise.
>>>>>>>>>
>>>>>>>>> How does your implementation handle this?
>>>>>>>>>
>>>>>>>>> I'd also like to hear from the other implementations here as to
>>>>>>>>> what they do and how their transaction manager implementations
>>>>>>>>> handle timeout.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> How would we word what the provider side has to do when detaching
>>>>>>>>>> entities after rollback [3]? I'm not sure that the
>>>>>>>>>> persistence provider will have the same chance to make a note for the
>>>>>>>>>> container side to take action on (if there is an
>>>>>>>>>> EE container involved). There is also the expectation that any JPA
>>>>>>>>>> provider will work, with any EE container to consider.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What do your transaction manager and container do?
>>>>>>>>>>>
>>>>>>>>>>>> Related to the above, if a JTA transaction rollback occurs in a
>>>>>>>>>>>> background thread [3], how are the managed entities
>>>>>>>>>>>> expected to be detached without violating the EntityManager
>>>>>>>>>>>> thread-safety [2]?
>>>>>>>>>>>>
>>>>>>>>>>>> There may be vendor specific solutions but shouldn't we (JPA spec eg)
>>>>>>>>>>>> account for the interaction of thread-unsafe
>>>>>>>>>>>> persistence contexts and the JTA Synchronization.afterCompletion that
>>>>>>>>>>>> may be invoked in non-application (background)
>>>>>>>>>>>> threads?
>>>>>>>>>>>>
>>>>>>>>>>>> Scott
>>>>>>>>>>>>
>>>>>>>>>>>> [1] 7.9.1 Container Responsibilities - After the JTA transaction has
>>>>>>>>>>>> completed (either by transaction commit or
>>>>>>>>>>>> rollback), the container closes the entity manager calling
>>>>>>>>>>>> EntityManager.close.
>>>>>>>>>>>>
>>>>>>>>>>>> [2] 7.2 Obtaining an EntityManager - An entity manager must not be
>>>>>>>>>>>> shared among multiple concurrently executing threads,
>>>>>>>>>>>> as the entity manager and persistence context are not required to be
>>>>>>>>>>>> threadsafe. Entity managers must only be accessed
>>>>>>>>>>>> in a single-threaded manner.
>>>>>>>>>>>>
>>>>>>>>>>>> [3] 7.9.2 Provider Responsibilities - When the JTA transaction rolls
>>>>>>>>>>>> back, the provider must detach all managed entities
>>>>>>>>>>>> if the persistence context is of type
>>>>>>>>>>>> SynchronizationType.SYNCHRONIZED
>>>>>>>>>>>> or has otherwise been joined to the transaction.
>>>>>>>>>>
>>>>>>>>>> [4] JTA 1.1 spec 3.4.3 Thread of Control:
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>> The X/Open XA interface specifies that the transaction association
>>>>>>>>>> related xa calls must be invoked from the same thread
>>>>>>>>>> context. This thread-of-control requirement is not applicable to the
>>>>>>>>>> object-oriented component-based application
>>>>>>>>>> run-time environment, in which application threads are dispatched
>>>>>>>>>> dynamically at method invocation time. Different Java
>>>>>>>>>> threads may be using the same connection resource to access the
>>>>>>>>>> resource manager if the connection spans multiple method
>>>>>>>>>> invocation. Depending on the implementation of the application server,
>>>>>>>>>> different Java threads may be involved with the
>>>>>>>>>> same XAResource object. The resource context and the transaction
>>>>>>>>>> context may be operated independent of thread context.
>>>>>>>>>> This means, for example, that it's possible for different threads to
>>>>>>>>>> be invoking the XAResource.start and XAResource.end
>>>>>>>>>> methods.
>>>>>>>>>>
>>>>>>>>>> If the application server allows multiple threads to use a single
>>>>>>>>>> XAResource object and the associated connection to the
>>>>>>>>>> resource manager, it is the responsibility of the application server
>>>>>>>>>> to ensure that there is only one transaction
>>>>>>>>>> context associated with the resource at any point of time.
>>>>>>>>>>
>>>>>>>>>> Thus the XAResource interface specified in this document requires that
>>>>>>>>>> the resource managers be able to support the
>>>>>>>>>> two-phase commit protocol from any thread context.
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>