Hello, Luis.
I see you are using GlassFish 3.1.2. We did make a change that seems to have helped with this particular error, and the fix is available in 3.1.2.2.
Is upgrading a possibility for you?
- Tim
On Jan 21, 2013, at 4:40 PM, Luis Pizarro wrote:
> Hi gentlemen,
>
> We keep finding this exception every time we stop/restart any instance - > "AMXStartupServiceNew.loadAMXMBeans: java.lang.IllegalStateException: BootAMX listener was not called"
>
> After doing some digging I believe that the instance is properly shut down. However, there seem to be some other processes happening right after that which cause the exception.
>
> I have browsed the web finding similar issues with no results. Hence I come here.
>
> This stack trace is coming from the instance's log file:
>
> [#|2013-01-21T15:37:12.692-0700|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=47;_ThreadName=Thread-2;|Server shutdown initiated|#]
>
> [#|2013-01-21T15:37:14.821-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,821 PWAssessment[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport LoopbackTransport
> |#]
>
> [#|2013-01-21T15:37:14.823-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,823 PWAssessment[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport JMSTransport
> |#]
>
> [#|2013-01-21T15:37:14.832-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,831 PWAssessment[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport HTTPTransport
> |#]
>
> [#|2013-01-21T15:37:14.870-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,870 PWBClinician[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport LoopbackTransport
> |#]
>
> [#|2013-01-21T15:37:14.874-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,874 PWBClinician[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport JMSTransport
> |#]
>
> [#|2013-01-21T15:37:14.881-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|2013-01-21 15:37:14,881 PWBClinician[-1] INFO MessagingDispatcher.java:233 MessagingDispatcher - Shutting down transport HTTPTransport
> |#]
>
> [#|2013-01-21T15:37:14.928-0700|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=48;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter stopping...|#]
>
> [#|2013-01-21T15:37:14.929-0700|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=48;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter stopped.|#]
>
> [#|2013-01-21T15:37:14.929-0700|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=49;_ThreadName=Thread-2;|RAR7094: jmsra shutdown successful.|#]
>
> [#|2013-01-21T15:37:16.722-0700|INFO|glassfish3.1.2|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=47;_ThreadName=Thread-2;|GMSAD1008: GMSAdapter for member: APOC-Assessment-n03x01 group: APOC-Assessment received GlassfishEventType: server_shutdown|#]
>
> [#|2013-01-21T15:37:16.724-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=47;_ThreadName=Thread-2;|GMS1096: member: APOC-Assessment-n03x01 is leaving group: APOC-Assessment|#]
>
> [#|2013-01-21T15:37:16.725-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=47;_ThreadName=Thread-2;|GMS1010: Leaving GMS group: APOC-Assessment with shutdown type set to InstanceShutdown|#]
>
> [#|2013-01-21T15:37:16.729-0700|CONFIG|glassfish3.1.2|ShoalLogger|_ThreadID=50;_ThreadName=Thread-2;|GMS1065: Completed processing outstanding master node messages for member: APOC-Assessment-n03x01 group: APOC-Assessment outstandingMessages to process: 0|#]
>
> [#|2013-01-21T15:37:16.732-0700|INFO|glassfish3.1.2|ShoalLogger.monitor|_ThreadID=47;_ThreadName=Thread-2;|BlockingIOMulicastSender monitoring stats: received: 30813 core poolsize:10 largest pool size:10 task count:30813 max queue size:0 rejected execution:0|#]
>
> [#|2013-01-21T15:37:16.732-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=51;_ThreadName=Thread-2;|GMS1110: Thread IP Multicast Listener for /228.9.158.231:13386 has completed.|#]
>
> [#|2013-01-21T15:37:16.894-0700|INFO|glassfish3.1.2|ShoalLogger.monitor|_ThreadID=47;_ThreadName=Thread-2;|GMS1115: router signal queue high water mark: 0 signal queue capacity: 600|#]
>
> [#|2013-01-21T15:37:16.893-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=52;_ThreadName=Thread-2;|GMS1088: MessageWindow thread for group: APOC-Assessment terminated due to shutdown notification|#]
>
> [#|2013-01-21T15:37:16.893-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=12;_ThreadName=Thread-2;|GMS1091: View Window event processing thread for group: APOC-Assessment terminated normally|#]
>
> [#|2013-01-21T15:37:16.894-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=53;_ThreadName=Thread-2;|GMS1107: SignalHandler task named GMS SignalHandler for Group-APOC-Assessment thread exiting|#]
>
> [#|2013-01-21T15:37:16.987-0700|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.server|_ThreadID=47;_ThreadName=Thread-2;|Initialized AMXStartupServiceNew in 27 ms, registered as amx-support:type=amx-loader,name=startup|#]
>
> [#|2013-01-21T15:37:17.096-0700|INFO|glassfish3.1.2|javax.enterprise.system.amx.org.glassfish.admin.amx.impl.mbean|_ThreadID=47;_ThreadName=Thread-2;|AMX024: Register children for instance name APOC-Assessment-n03x01|#]
>
> [#|2013-01-21T15:37:17.110-0700|INFO|glassfish3.1.2|javax.enterprise.system.amx.org.glassfish.admin.amx.impl.mbean|_ThreadID=47;_ThreadName=Thread-2;|AMX020: AMX ComplianceMonitor: ValidationLevel = full, UnregisterNonCompliant = false, LogInaccessibleAttributes = true|#]
>
> [#|2013-01-21T15:37:17.147-0700|INFO|glassfish3.1.2|javax.enterprise.system.amx.org.glassfish.admin.amx.impl.config|_ThreadID=54;_ThreadName=Thread-2;|AMX012: In AMXConfigLoader : Loading null|#]
>
> [#|2013-01-21T15:37:18.089-0700|INFO|glassfish3.1.2|javax.enterprise.system.amx.org.glassfish.admin.amx.impl.config|_ThreadID=54;_ThreadName=Thread-2;|AMX013: AMX domain config registered as amx:pp=/,type=domain|#]
>
> [#|2013-01-21T15:37:18.346-0700|INFO|glassfish3.1.2|javax.enterprise.system.amx.org.glassfish.admin.amx.impl.j2ee.loader|_ThreadID=55;_ThreadName=Thread-2;|AMX014: J2EEDomain registered at amx:pp=/,type=J2EEDomain,j2eeType=J2EEDomain,name=amx|#]
>
> [#|2013-01-21T15:37:18.349-0700|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.server|_ThreadID=47;_ThreadName=Thread-2;|AMXStartupServiceNew: AMX ready for use, DomainRoot = amx:pp=,type=domain-root|#]
>
> [#|2013-01-21T15:37:18.350-0700|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=47;_ThreadName=Thread-2;|AMXStartupServiceNew.loadAMXMBeans: java.lang.IllegalStateException: BootAMX listener was not called|#]
>
> [#|2013-01-21T15:37:18.352-0700|WARNING|glassfish3.1.2|javax.enterprise.system.core.org.glassfish.kernel.event|_ThreadID=47;_ThreadName=Thread-2;|Exception while dispatching an event
> javax.management.RuntimeMBeanException: java.lang.RuntimeException: java.lang.IllegalStateException: BootAMX listener was not called
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at org.glassfish.admin.mbeanserver.BootAMX.bootAMX(BootAMX.java:163)
> at org.glassfish.admin.mbeanserver.AdminAuthorizedMBeanServer$Handler.amxDomain(AdminAuthorizedMBeanServer.java:154)
> at org.glassfish.admin.mbeanserver.AdminAuthorizedMBeanServer$Handler.isAMX(AdminAuthorizedMBeanServer.java:149)
> at org.glassfish.admin.mbeanserver.AdminAuthorizedMBeanServer$Handler.isAMX(AdminAuthorizedMBeanServer.java:142)
> at org.glassfish.admin.mbeanserver.AdminAuthorizedMBeanServer$Handler.isAllowed(AdminAuthorizedMBeanServer.java:136)
> at org.glassfish.admin.mbeanserver.AdminAuthorizedMBeanServer$Handler.invoke(AdminAuthorizedMBeanServer.java:101)
> at $Proxy185.unregisterMBean(Unknown Source)
> at org.glassfish.admin.mbeanserver.JMXStartupService$JMXConnectorsStarterThread.shutdown(JMXStartupService.java:239)
> at org.glassfish.admin.mbeanserver.JMXStartupService.shutdown(JMXStartupService.java:160)
> at org.glassfish.admin.mbeanserver.JMXStartupService.access$000(JMXStartupService.java:94)
> at org.glassfish.admin.mbeanserver.JMXStartupService$ShutdownListener.event(JMXStartupService.java:128)
> at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:128)
> at com.sun.enterprise.v3.server.AppServerStartup.stop(AppServerStartup.java:439)
> at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.stop(GlassFishImpl.java:88)
> at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.stop(GlassFishDecorator.java:68)
> at com.sun.enterprise.v3.admin.StopServer.doExecute(StopServer.java:70)
> at com.sun.enterprise.v3.admin.cluster.StopInstanceInstanceCommand.execute(StopInstanceInstanceCommand.java:94)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.run(CommandRunnerImpl.java:377)
> Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: BootAMX listener was not called
> at org.glassfish.admin.amx.impl.AMXStartupService.loadAMXMBeans(AMXStartupService.java:247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at javax.management.StandardMBean.invoke(StandardMBean.java:391)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> ... 20 more
> Caused by: java.lang.IllegalStateException: BootAMX listener was not called
> at org.glassfish.admin.amx.impl.AMXStartupService._loadAMXMBeans(AMXStartupService.java:376)
> at org.glassfish.admin.amx.impl.AMXStartupService.loadAMXMBeans(AMXStartupService.java:244)
> ... 31 more
> |#]
>
> [#|2013-01-21T15:37:18.355-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=47;_ThreadName=Thread-2;|Shutdown procedure finished|#]
>
>
>
> This stack trace is coming from the domain's log file:
>
> [#|2013-01-21T15:37:12.598-0700|INFO|glassfish3.1.2|org.glassfish.admingui|_ThreadID=121;_ThreadName=Thread-2;|https://localhost:4848/management/domain/servers/server/APOC-Assessment-n03x01/stop-instance|#]
>
> [#|2013-01-21T15:37:12.608-0700|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=122;_ThreadName=Thread-2;|Instance APOC-Assessment-n03x01 shutdown initiated|#]
>
> [#|2013-01-21T15:37:16.720-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=25;_ThreadName=Thread-2;|GMS1092: GMS View Change Received for group: APOC-Assessment : Members in view for PEER_STOP_EVENT(before change analysis) are :
> 1: MemberId: APOC-Assessment-n01x01, MemberType: CORE, Address: 192.168.10.81:9140:228.9.158.231:13386:APOC-Assessment:APOC-Assessment-n01x01
> 2: MemberId: APOC-Assessment-n02x01, MemberType: CORE, Address: 192.168.10.82:9200:228.9.158.231:13386:APOC-Assessment:APOC-Assessment-n02x01
> 3: MemberId: server, MemberType: SPECTATOR, Address: 192.168.10.81:9119:228.9.158.231:13386:APOC-Assessment:server
> |#]
>
> [#|2013-01-21T15:37:16.720-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=25;_ThreadName=Thread-2;|GMS1016: Analyzing new membership snapshot received as part of event: PEER_STOP_EVENT for member: APOC-Assessment-n03x01 of group: APOC-Assessment|#]
>
> [#|2013-01-21T15:37:16.720-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=25;_ThreadName=Thread-2;|GMS1017: Received PlannedShutdownEvent Announcement from member: APOC-Assessment-n03x01 with shutdown type: INSTANCE_SHUTDOWN of group: APOC-Assessment|#]
>
> [#|2013-01-21T15:37:16.720-0700|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=111;_ThreadName=Thread-2;|GMS1008: Sending PlannedShutdownSignals to registered Actions for shutdownType INSTANCE_SHUTDOWN member: APOC-Assessment-n03x01 ...|#]
>
> Thank you in advance.
>
> Luis