users@glassfish.java.net

Re: JMS issue

From: Danilo Levantesi <danilo.levantesi_at_gmail.com>
Date: Tue, 6 Oct 2009 18:41:33 +0200

Hi,
many thanks for your reply.
Google returned me this: http://weblogs.java.net/blog/felipegaucho/archive/2009/09/24/handling-poison-messages-glassfish

I double-reviewed the code, ensuring that the onMessage() doesn't throw any exception, with a TransactionAttribute.REQUIRED annotation "protecting" the method.
By the way, the architecture was "correct" (or at least followed the above structure), so the problem persist.

Struggling with this issue, I noticed that the machine (it is a xen guest) has many clock skews during the day, and the time is set +/- 1 hour.
So I made a simple client that posts TextMessages on a Topic; during the process the clock is changed (+/- 1hour randomly), and it reproduced the issue.

Why does this happen? Is the JMS protocol vulnerable to clock skew?
Moreover, why do this issue leads to a total Glassfish crash? I have to "kill -9" it in order to restart...

Many thanks

> Look my blog about "handling poison messages with Glassfish"...
>
> ... I am typing from the phone so I cannot google the link ;)
>
> On 02.10.2009, at 18:11, Danilo Levantesi <danilo.levantesi_at_gmail.com>
> wrote:
>
> > Hi to all,
> > I'm experiencing a weird issue with JMS.
> >
> > I'm using Glassfish v2.1, where an EAR is deployed. One of the EJBs
> > contains is a JMS topic producer and 3 MDBs are the subscribers.
> > The JMS Service is configured to run with LOCAL mode; the other
> > options are the defaults one.
> >
> > Often it happens that I found the whole server freezed; looking for
> > a cause, I found the IMQ brooker unresponsive.
> > The imqcmd command cannot connect to the local instance.
> > The Glassfish logs show many message like the following:
> >
> > [#|2009-10-02T15:39:55.986+0200|WARNING|sun-appserver2.1|
> > javax.jms.connection|_ThreadID=29;_ThreadName=p: thread-pool-1; w: 1
> > 228;_RequestID=31f2e817-d644-4cc1-9ce8-3dfd8fe2b32e;|[W2003]: Broker
> > not responding [START_TRANSACTION(44)] for 120 seconds.
> > Still trying..., broker addr=localhost:7678(36909),
> > connectionID=345409444345888768, clientID=null, consumerID=337626|#]
> >
> > The imq logs, instead, show many of the following:
> >
> > [02/ott/2009:15:52:01 CEST] WARNING [B2176]: Force
> > [CONNECTION_CLEANUP] ended (TMFAIL) transaction
> > TID=345409472186151168[STA
> > RTED(1)],
> > XID=
> > 77726F70312E6D6178616D692E6E65742C7365727665722C50333730322C00D5E20300D61C120F77726F70312E6D6178616D692E6E65742
>
>
> > C7365727665722C5033373032
> >
> > [02/ott/2009:15:52:01 CEST] WARNING [B2177]: Force
> > [CONNECTION_CLEANUP] rolled back transaction TID=345409472186151168
> > [STARTE
> > D(1)],
> > XID=
> > 77726F70312E6D6178616D692E6E65742C7365727665722C50333730322C00D5E20300D61C120F77726F70312E6D6178616D692E6E65742C73
>
>
> > 65727665722C5033373032
> >
> > and then:
> >
> > [02/ott/2009:15:58:01 CEST] [B2173]: Destroying unresponsive
> > connection 345409444349025280, connection has not responded for
> > 933 seconds
> > [02/ott/2009:15:58:01 CEST] [B1066]: Closing:
> > guest_at_127.0.0.1:46984->jms:36909 because "Connection unresponsive".
> > Count: se
> > rvice=0 broker=5
> > [02/ott/2009:15:58:01 CEST] Internal Warning: message PING(54):
> > 0-0.0.0.0-0-0queued on destroyed connection IMQConn[DESTROYED,
> > guest_at_127.0.0.1:46984,jms:36909]
> >
> >
> > I cannot find a starting point to debug the cause: the same
> > application runs on two different machines, and only one of them
> > shows the above errors.
> > Moreover the issue appears randomly: sometimes the server runs
> > smoothly for one or two days, sometimes only for few minutes.
> >
> > Many thanks in advance
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>