dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Prashanth Abbagani <Prashanth.Abbagani_at_Sun.COM>
Date: Thu, 16 Jul 2009 11:33:54 -0700

Prashanth Abbagani wrote:
> Alexis Moussine-Pouchkine wrote:
>> thanks Prashanth, this is useful.
>>
>> Note that if monitoring is not enabled, the requests fail with an NPE
>> (promoted build 55):
> Doesn't seem to happen in the monitoring infra. Can I see your
> provider class, is this checked in?
>>
>>> SEVERE: service exception
>>> java.lang.NullPointerException
>>> at
>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>
>>> at
>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>
>>> at
>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>
>>> at
>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:87)
>>>
>>> at
>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:61)
>>>
>>> at
>>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:254)
>>>
>>> at
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>
>>> at
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>
>>> at
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>
>>> at
>>> com.sun.jersey.server.impl.container.grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>
>>> at
>>> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:165)
>>>
>>> at
>>> com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
>>>
>>> at
>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:208)
>>>
>>> at
>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>
>>> at
>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
>>> at
>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
>>> at
>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
>>>
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
>>>
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
>>>
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
>>>
>>> at
>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
>>>
>>> at
>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>
>>> at
>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>
>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>
>>> at java.lang.Thread.run(Thread.java:637)
>>
>> An error message suggesting to turn on monitoring would be nice.
> I agree, I need to understand at what point is this happening. If this
> is in the ProbeProvider, the monitoring flag has nothing to do with this.
>>
>> Also, are core classes monitoring loaded prior to any call?
>> I see several messages at startup (even without the javaagent option)
>> which seem to have an impact on startup time.
> The Providers are discovered by default during the module's startup.
> The retrasformation of ProbeProvider (which is the only
> compute-intensive operation) will only happen when there are listeners
> (StatsProviders) registered and these statsProviders are registered
> when the corresponding monitoring flag is turned on. So there is no
> overhead when the monitoring is turned off. We do have the Monitoring
> startup class which keeps discovering the providers as they come (even
> without the javaagent), this is a required step and I dont think it
> effects the startup time significantly.
I stand corrected. When the Btrace agent is disabled (explained in bug
<https://glassfish.dev.java.net/issues/show_bug.cgi?id=8706>), theres is
no reason to discover and process the providers, we are planning to omit
this step so there is no startup overhead at all for monitoring
>>
>> thanks,
>> -Alexis
>>
>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>
>>> Hello,
>>>
>>> The monitoring in V3 is now integrated and working with several
>>> stats from various module showing up. See the instructions towards
>>> the end to which explain on how to view these stats.
>>>
>>> Following are the features of monitoring that were accomplished
>>> before the soft-code freeze deadline.
>>> • Monitoring Probe Providers bundled by the module owners are
>>> discovered automatically and processed so probes are being fired
>>> when it visits module's code path
>>> • Following is done with the monitoring StatsProviders which
>>> listens to the probes and updates the statistics
>>> • We process the statsProviders (annotations) for the stats
>>> you have annotated to be exposed
>>> • Exposing the stats through CLI (asadmin)
>>> • Exposing the stats with REST
>>> • GUI team is working on exposing these stats for some of
>>> the critical modules
>>> • Integrate with Gmbal to expose these stats as JMX/AMX MBeans
>>> • We also are ensuring backward compatibility for all the V2
>>> level stats
>>> • Handling of config changes (from CLI,GUI or REST), so the
>>> stats get enabled/disabled automatically
>>> • We are using BTrace as our underlying infrastructure to do the
>>> byte-code instrumentation of the provider classes so the probes are
>>> fired and the registered listeners (StatsProviders) are called.
>>> Steps to see the monitoring in action
>>>
>>> You will need the BTrace agent running. You will need to start the
>>> server with the following agent option (See Issue 8706, you
>>> shouldn't need this step once its fixed.)
>>>
>>> $>java
>>> -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=dumpClasses=true,unsafe=true
>>> -jar glassfishv3/glassfish/modules/glassfish.jar
>>>
>>> To see the monitoring stats, you will have to turn on monitoring for
>>> the interested modules (see the attached script to turn them all to ON)
>>>
>>> There are various ways to see your stats,
>>> • REST - http://localhost:4848/monitoring/domain
>>> • asadmin - ./asadmin get --monitor=true "*"
>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>
>>> Let us know if you run into any issues or if you have any questions.
>>>
>>> thanks
>>> Prashanth
>>>
>>>
>>> #!/bin/sh
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
>>>
>>> ./asadmin set
>>> configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>