From: Edward Bratt <ed.bratt_at_oracle.com>
Date: Wed, 02 Mar 2011 13:45:58 -0800

Wanted to complete this post --
We found that the broker was exhausting memory after loading several hundred thousand messages. When the broker encounters this condition, it does NOT remove the messages from the persistent store. However, it cannot load them either so, the messages are not available and the broker continues on, attempting to provide service anyway. This condition was indicated in the Broker log.

We did identify that the exception was not properly propagated back to the server and, we have created an issue to track resolution of this secondary problem (http://java.net/jira/browse/MQ-77).

Our advice was to increase the heap allocation and that should allow the broker to start, as well as to recover all the messages stored on disk. We also advised the user to consider applying destination limits, to avoid similar situations that might prevent restart, in the future.

If this message is seen, in the MQ broker logs:

[07/Fev/2011:09:10:58 BRST] ERROR [B2085]: Loading Destination EVENT_IN [Queue] failed. Messages stored on that destination will not be available.:

java.lang.OutOfMemoryError: Java heap space

Try increasing the heap and starting the broker again.

The persistent store is not purged when this exception is encountered. Further, unless the user takes overt action to purge a destination, destinations are not deleted when redeploying MDBs, or any other Java EE artifact.

In case anyone is interested, the broker only loads indexing information for each message, not the entire message body. An additional complication to this issue was that the user was also using a profiling tools which may have also been competing for resources with the broker.

-- Ed Bratt

on 2/10/2011 1:50 PM Edward Bratt said the following:

We don't believe this should be happening but perhaps you can give a bit more detail about what commands you actually ran.
First, can you confirm that the destination is a Queue and not a Topic? Topics could certainly behave this way, even if they are persistent.
If you have the logs from both the MQ broker and, from the GlassFish servers, we'd like to review them. Preferably across the redeployment operations.
If you can send us the complete list of commands you ran, to deploy/undeploy/configure/reconfigure your instances and cluster, we should be able to get a better answer for you.
If the log files are large, or you would prefer not sending them to the full list, you can e-mail me directly (ed.bratt@oracle.com). I'll make sure the right developer(s) get them for review.
Thanks,
-- Ed

On 2/9/2011 2:13 AM, marciogj@gmail.com wrote:

Hi,

I´m interested to understand how Glassfish and OpenMQ works together
since we started to experience a odd situation:
We have a persistent queue which receives quite a lot of messages per
second.
After a while we had 350.000 messages in the queue (Almost 60MB). At
this point, we did a simple undeploy and deploy of our .ear application
(without any source code change).

The result of this operation was catastrophic: a Java heap space during
deploy and worst of all - The queue was cleaned!
This is our top requirement, never lose a single message. In this case
we lost million application messages since each jms message contains a
batch from our application.

I cannot understand such situation since we use persistent messages.
It's looks like Glassfish loads all message into a HashMap but I´m not
sure (I´m guessing by java heap error)...

The question is why Glassfish and Open MQ would touch data from a
persistent queue during application load? And why delete all messages
without any reasonable reason?

Other details:
Glassfish v2.1.1
Queue FlowLimit = 1

Regards