Hello,
we are currently running a 2 node cluster on glassfish 3.1.2.2 with JMS (OpenMQ 4.5.2 Patch 2) running in embedded mode and a shared database (MySQL Percona Cluster). This worked fine until this morning. We restarted both instances because of a kernel update. Further we updated some software packages (java 1.7.0.19 -> 1.7.0.25) and a new minor release of percona.
Then when restarting our nodes. One node had the following error:
java.lang.NoClassDefFoundError: com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl
We found this class in the following file:
cd /opt/glassfish3/mq/lib
jar -tf imq.jar | grep com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl
It turns out when removing the STOMP bridge from the JMS broker the message dissappears but also stomp dissappears which we are heavily using.
/opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/props/config.properties
imq.bridge.enabled=true
imq.bridge.activelist=stomp
imq.bridge.admin.user=admin
imq.bridge.admin.password=*****
/opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/props/config.properties
[#|2013-07-03T15:16:33.949+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=140;_ThreadName=Thread-2;|JMS011: Addresslist mq://10.39.51.118:27676/|#]
[#|2013-07-03T15:16:33.951+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=140;_ThreadName=Thread-2;|JMS011: Addresslist mq://10.239.75.9:27676/|#]
[#|2013-07-03T15:16:33.961+0000|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.recovery|_ThreadID=140;_ThreadName=Thread-2;|JMS resource recovery has created CFs = 2|#]
[#|2013-07-03T15:16:33.991+0000|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=140;_ThreadName=Thread-2;|Exception in thread "Recovery Helper Thread" |#]
[#|2013-07-03T15:16:34.010+0000|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=140;_ThreadName=Thread-2;|java.lang.NoClassDefFoundError: com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl
at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:97)
at com.sun.messaging.jms.ra.ManagedConnection.<init>(ManagedConnection.java:196)
at com.sun.messaging.jms.ra.ManagedConnectionFactory.createManagedConnection(ManagedConnectionFactory.java:226)
at com.sun.enterprise.resource.recovery.ConnectorsRecoveryResourceHandler.loadXAResourcesAndItsConnections(ConnectorsRecoveryResourceHandler.java:292)
at com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.getAllRecoverableResources(ResourceRecoveryManagerImpl.java:210)
at com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.recoverXAResources(ResourceRecoveryManagerImpl.java:234)
at com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.recoverXAResources(ResourceRecoveryManagerImpl.java:331)
at com.sun.enterprise.transaction.jts.ResourceRecoveryManagerImpl.postConstruct(ResourceRecoveryManagerImpl.java:106)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at org.jvnet.hk2.component.Habitat.getByContract(Habitat.java:1050)
at com.sun.jts.jta.TransactionServiceProperties$RecoveryHelperThread.run(TransactionServiceProperties.java:358)
|#]
/opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/bridges/stomp/stomp0.log
Jul 03, 2013 1:52:35 PM
INFO: STOMP server log domain: imq.bridge.stomp
Jul 03, 2013 1:52:35 PM
INFO: STOMP server log file: /opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/bridges/stomp/stomp%g.log[0,1]
Jul 03, 2013 1:52:35 PM
INFO: Initialize Grizzly NIO framework 1.9.50
Jul 03, 2013 1:52:35 PM
INFO: Start STOMP server TCP transport on ip-10-39-51-118.eu-west-1.compute.internal:7672
Jul 03, 2013 1:52:35 PM
INFO: STOMP server has been successfully started with -1 reader threads
Jul 03, 2013 1:55:01 PM
INFO: Create JMS connection for user admin
Jul 03, 2013 1:55:01 PM
SEVERE: Could not initialize class com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl
java.lang.NoClassDefFoundError: Could not initialize class com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl
at com.sun.messaging.BasicConnectionFactory.createConnection(BasicConnectionFactory.java:153)
at com.sun.messaging.bridge.service.stomp.StompConnection.connect(StompConnection.java:106)
at com.sun.messaging.bridge.service.stomp.StompProtocolHandler.onCONNECT(StompProtocolHandler.java:148)
at com.sun.messaging.bridge.service.stomp.StompProtocolFilter.execute(StompProtocolFilter.java:107)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:724)
Jul 03, 2013 1:55:01 PM
SEVERE: [BSS3002]: SUBSCRIBE failed: [BSS4009]: Not connected ! on STOMP connection
com.sun.messaging.bridge.service.stomp.NotConnectedException: [BSS4009]: Not connected !
at com.sun.messaging.bridge.service.stomp.StompConnection.checkConnection(StompConnection.java:425)
at com.sun.messaging.bridge.service.stomp.StompConnection.createSubscriberSession(StompConnection.java:312)
at com.sun.messaging.bridge.service.stomp.StompProtocolHandler.onSUBSCRIBE(StompProtocolHandler.java:417)
at com.sun.messaging.bridge.service.stomp.StompProtocolFilter.execute(StompProtocolFilter.java:118)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:724)
Jul 03, 2013 1:55:01 PM
SEVERE: [BSS3002]: UNSUBSCRIBE failed: [BSS4009]: Not connected ! on STOMP connection
com.sun.messaging.bridge.service.stomp.NotConnectedException: [BSS4009]: Not connected !
at com.sun.messaging.bridge.service.stomp.StompConnection.checkConnection(StompConnection.java:425)
at com.sun.messaging.bridge.service.stomp.StompConnection.checkConnection(StompConnection.java:413)
at com.sun.messaging.bridge.service.stomp.StompConnection.closeSubscriber(StompConnection.java:349)
at com.sun.messaging.bridge.service.stomp.StompProtocolHandler.onUNSUBSCRIBE(StompProtocolHandler.java:491)
at com.sun.messaging.bridge.service.stomp.StompProtocolFilter.execute(StompProtocolFilter.java:121)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:724)
Jul 03, 2013 1:55:01 PM
SEVERE: [BSS3002]: DISCONNECT failed: [BSS4009]: Not connected ! on STOMP connection
Jul 03, 2013 3:04:40 PM
INFO: STOMP server log domain: imq.bridge.stomp
Jul 03, 2013 3:04:40 PM
INFO: STOMP server log file: /opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/bridges/stomp/stomp%g.log[0,1]
Jul 03, 2013 3:04:40 PM
INFO: Initialize Grizzly NIO framework 1.9.50
Jul 03, 2013 3:04:41 PM
INFO: Start STOMP server TCP transport on ip-10-39-51-118.eu-west-1.compute.internal:7672
Jul 03, 2013 3:04:41 PM
INFO: STOMP server has been successfully started with -1 reader threads
Jul 03, 2013 3:15:20 PM
INFO: STOMP server log domain: imq.bridge.stomp
Jul 03, 2013 3:15:20 PM
INFO: STOMP server log file: /opt/glassfish3/glassfish/nodes/node004.apaxo.de/1.004.apaxo.de/imq/instances/SemRecSys1004apaxode/bridges/stomp/stomp%g.log[0,1]
Jul 03, 2013 3:15:20 PM
INFO: Initialize Grizzly NIO framework 1.9.50
Jul 03, 2013 3:15:20 PM
INFO: Start STOMP server TCP transport on ip-10-39-51-118.eu-west-1.compute.internal:7672
Jul 03, 2013 3:15:21 PM
INFO: STOMP server has been successfully started with -1 reader threads
Is there someone who knows what the problem might be?
Thanks
Manuel
--
Manuel Blechschmidt
M.Sc. IT Systems Engineering
CTO - Apaxo GmbH
blechschmidt_at_apaxo.de
http://www.apaxo.de
Weinbergstr. 16
14469 Potsdam
Telefon +49 (0)6204 9180 593
Fax +49 (0)6204 9180 594
Mobil: +49 173/6322621
Skype: Manuel_B86
Twitter: http://twitter.com/Manuel_B
Sitz der Gesellschaft: Viernheim
Handelsregister HRB 87159
Ust-IdNr. DE261368874
Amtsgericht Darmstadt
Geschäftsführer Friedhelm Scharhag