Paul,
> I removed the @PostConstruct and @PreConstruct methods from
QuestionManagerBean as well as any usage of the
QuestionManagerBean.announceQuestion() function. The error does not
occur in this configuration!
Which error ? The one in your first email for EMBEDDED mode or the
second one in LOCAL mode ?
@PostConstruct is to create the JMS connection, what does @PreConstruct
and QuestionManagerBean.announceQuestion() do (not shown in your code
snippet below) ? or did you actually mean @PreDestroy which is to close
the JMS connection and QuestionManagerBean.announceNewQuestion which is
to send a JMS message ? If the later, the problem looks like a JMS
related issue if without calling these methods the problem does not
occur. It's possible the problem is triggered by
create/closeConnection, which could indicates a JMS related bug
(GlassFish+JMSRA) in the area of "recycle" JMS connection. You can try
to set the bean pool configuration of QuestionManagerBean to avoid bean
destory, e.g. no idle timeout, max pool size large enough for your
possible highest load and do the similar for the JMS connector pool.
However, it's necessary to find out the root cause the problem in order
to give you the right advise to avoid (if possible) the problem, and
most importantly to ensure the problem is fixed in a later release of
GlassFish.
How is QuestionManagerBean.announceNewQuestion invoked ?
Could you please file a JIRA issue for this with as much information as
possible in order to reproduce it (preferrably with a reproducible test
case, and be sure to include GlassFish version/build #) ?
Thanks,
amy
On 12/28/2010 10:22 AM, Paul Giblock wrote:
> Amy -
>
> Thank you for the assistance. I have not tried GF3.1b34 yet,
> primarily because the issue only affects both of our production
> servers and I do not want to change the configuration too much.
> However, I have discovered the underlying cause of the problem and I
> would like some advice regarding how to fix it.
>
> The WidgetHelperBean class, the one containing the intializeWidget()
> method seen in the stacktrace, is structured as such (symbols
> abbreviated to protect the mailing-list ). This class has no JMS
> resources and doesn't do anything special with JMS or JTA:
>
> @Stateless
> @PersistenceContext(unitName = "vhm", name = "VhmEM")
> public class WidgetHelperBean implements WidgetHelper {
> @EJB QuestionManager questionMgr;
>
> @Override
> public WidgetInitInfo initializeWidget (long e, Class c, Long ci,
> boolean m, int s, String ip) {
> // No programmatic JMS or JTA stuff at all
> // ...
> questionMgr.listUserQuestionsForEvent(e, QuestionListSort.POPULAR,
> 0, Math.max(initSize, INITIAL_PAGE_SIZE));
> // ...
> }
> }
>
>
> The @EJB is injected from a local bean that looks like:
>
> @Stateless
> @PersistenceContext(unitName = "vhm", name = "VhmEM")
> public class QuestionManagerBean implements QuestionManager {
>
> @Resource(name="ConnectionFactory",
> mappedName="ConnectionFactory")
> private ConnectionFactory connectionFactory;
> private Connection connection;
>
> @Resource(name=NewContent.TOPIC,
> mappedName= NewContent.TOPIC)
> private Topic newContentTopic;
>
> @PostConstruct
> public void makeConnection() {
> connection = JmsUtils.createConnection(connectionFactory);
> }
>
> @PreDestroy
> public void endConnection() {
> JmsUtils.closeConnection(connection);
> }
>
> // Called by WidgetHelperBean.initializeWidget above
> @TransactionAttribute(TransactionAttributeType.SUPPORTS)
> public UserQuestionList listUserQuestionsForEvent (
> long eventId, QuestionListSort sort) {
> // Loads data from JPA, NO calls to JMS or JTA
> }
>
> // Not called by WidgetHelperBean or by ANY methods within that TX
> private void announceNewQuestion (Question question) {
> Session session = null;
> try {
> session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
> MessageProducer producer = session.createProducer(newContentTopic);
> ObjectMessage msg = session.createObjectMessage();
>
> // Send msg
> if (msg != null && producer != null) {
> msg.setObject(question);
> msg.setStringProperty(NewContent.TYPE, Question.TYPE);
> msg.setIntProperty(NewContent.CAUSE, NewContent.CAUSE_SUBMITTED);
> producer.send(msg);
> }
> }
> catch (JMSException je) {
> throw new IllegalStateException(
> "MessageProducer could not be created or send message", je);
> }
> catch (Exception ex) {
> log.error("Something terrible happened", ex);
> }
> finally {
> VHMFileUtils.safeClose(session);
> }
> }
>
> // ...
> }
>
>
> For completion sake, here are the 3 utility functions used in the
> previous code snippet:
>
> public class JmsUtils {
> private static final Logger log =
> LoggerFactory.getLogger(JmsUtils.class);
>
> public static Connection createConnection (ConnectionFactory factory) {
> final String msg = "Could not connect to JMS.";
> try {
> return factory.createConnection();
> }
> catch (JMSException e) {
> log.error(msg, e);
> throw new IllegalStateException(msg, e);
> }
> }
>
> public static void closeConnection (Connection connection) {
> final String msg = "Could not close JMS connection";
> if (connection != null) {
> try {
> connection.close();
> }
> catch (JMSException e) {
> log.error(msg);
> throw new IllegalStateException(msg, e);
> }
> }
> }
> }
>
>
> public class VHMFileUtils {
> static final Logger log = LoggerFactory.getLogger(VHMFileUtils.class);
> public static void safeClose (Session s) {
> try {
> if (s != null) {
> s.close();
> }
> }
> catch (Throwable t) {
> log.error("Failed to close session!", t);
> }
> }
> }
>
>
> Anyways, there you have it. The error only happens on our production
> systems and it doesn't happen consistently. I removed the
> @PostConstruct and @PreConstruct methods from QuestionManagerBean as
> well as any usage of the QuestionManagerBean.announceQuestion()
> function. The error does not occur in this configuration!
>
> Hooray! So my guess is that our production system creates more
> instances of the WidgetHelperBean (and by extension, the
> QuestionManagerBean as well) due to higher load. This in turn causes
> more JMS connections to be created. Can there be too many JMS
> connections? I'm following the recipe given by Sun for EJB2+JMS and
> presented in "EJB3 In Action" by Panda, et al.
>
> What is the solution? Do we need to ensure that the
> ConnectionFactory's max-pool-size is at least as large as
> number-of-distinct-ejbs * ejb-max-pool-size? Or, should we only
> create connections on-demand (within the
> QuestionManagerBean.announceQuestion method)? Or, is something just
> terribly wrong? Certainly, the problem exposes itself in a very
> non-intuitive way.
>
> Anyways, I hope this helps, and I hope you will have some advice. If
> you need anything else (Config files, system information, etc..)
> please let me know.
>
> Thank you,
> Paul G