dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Prashanth Abbagani <Prashanth.Abbagani_at_Sun.COM>
Date: Thu, 16 Jul 2009 20:51:14 -0700

Rajeshwar Patil wrote:
> Alexis Moussine-Pouchkine wrote:
>>
>> On Jul 16, 2009, at 22:51, Rajeshwar Patil wrote:
>>
>>> Prashanth Abbagani wrote:
>>>> Alexis Moussine-Pouchkine wrote:
>>>>>
>>>>> On Jul 16, 2009, at 19:12, Prashanth Abbagani wrote:
>>>>>
>>>>>> Alexis Moussine-Pouchkine wrote:
>>>>>>> thanks Prashanth, this is useful.
>>>>>>>
>>>>>>> Note that if monitoring is not enabled, the requests fail with
>>>>>>> an NPE (promoted build 55):
>>>>>> Doesn't seem to happen in the monitoring infra. Can I see your
>>>>>> provider class, is this checked in?
>>>>>
>>>>> Sorry I should have been more detailed.
>>>>> Simply start b55 (monitoring OFF) and hit
>>>>> http://localhost:4848/monitoring/domain/server/jvm/runtime
>>>>> It should trigger the exception below.
>>>>> -Alexis
>>>> I will give it a try, but this is happening in rest, the underlying
>>>> problem could be Monitoring Infra. Rajeshwar, can you pls debug the
>>>> problem and let me know the root cause here.
>>> Hello Prashanth,
>>> Monitoring infrastructure is returning attribute list (nodes with
>>> null value) even when monitoring if OFF. Regardless, I will also
>>> put a null check in REST code.
>>
>> Hi Rajeshwar,
>>
>> Is there anyway that situation could be translated into a message to
>> the user saying that he should turn monitoring to HIGH (or LOW) if
>> he's trying to monitoring GF using the REST interface?
> Hello Alexis,
> I believe user should never see top level JVM node itself if
> monitoring is OFF for JVM.
This is a bug in monitoring. I created an issue
<https://glassfish.dev.java.net/issues/show_bug.cgi?id=8751> and
assigned to myself. This should be fixed soon.
> We can not do provide message in a generic way. We can do this for
> known or default modules but it won't be possible for new/third party
> modules which can bring in new monitoring nodes with their own
> configuration flags. Being said this, I can see whether I can provide
> massage for the top level resource
> (http://localhost:port/management/domain) when no child node is
> available at all. This is the most likely use case user will run into.
>
> Thanks
> Rajeshwar
>>
>> thanks!
>> -Alexis
>>
>>>
>>> Thanks
>>> Rajeshwar
>>>>
>>>> thanks
>>>> Prashanth
>>>>>
>>>>>>>
>>>>>>>> SEVERE: service exception
>>>>>>>> java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:87)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:61)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:254)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.jersey.server.impl.container.grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:165)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:208)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
>>>>>>>> at
>>>>>>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>>>>>
>>>>>>>> at
>>>>>>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>>>>>>
>>>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>
>>>>>>>> at java.lang.Thread.run(Thread.java:637)
>>>>>>>
>>>>>>> An error message suggesting to turn on monitoring would be nice.
>>>>>> I agree, I need to understand at what point is this happening. If
>>>>>> this is in the ProbeProvider, the monitoring flag has nothing to
>>>>>> do with this.
>>>>>>>
>>>>>>> Also, are core classes monitoring loaded prior to any call?
>>>>>>> I see several messages at startup (even without the javaagent
>>>>>>> option) which seem to have an impact on startup time.
>>>>>> The Providers are discovered by default during the module's
>>>>>> startup. The retrasformation of ProbeProvider (which is the only
>>>>>> compute-intensive operation) will only happen when there are
>>>>>> listeners (StatsProviders) registered and these statsProviders
>>>>>> are registered when the corresponding monitoring flag is turned
>>>>>> on. So there is no overhead when the monitoring is turned off. We
>>>>>> do have the Monitoring startup class which keeps discovering the
>>>>>> providers as they come (even without the javaagent), this is a
>>>>>> required step and I dont think it effects the startup time
>>>>>> significantly.
>>>>>>>
>>>>>>> thanks,
>>>>>>> -Alexis
>>>>>>>
>>>>>>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> The monitoring in V3 is now integrated and working with several
>>>>>>>> stats from various module showing up. See the instructions
>>>>>>>> towards the end to which explain on how to view these stats.
>>>>>>>>
>>>>>>>> Following are the features of monitoring that were accomplished
>>>>>>>> before the soft-code freeze deadline.
>>>>>>>> • Monitoring Probe Providers bundled by the module owners are
>>>>>>>> discovered automatically and processed so probes are being
>>>>>>>> fired when it visits module's code path
>>>>>>>> • Following is done with the monitoring StatsProviders which
>>>>>>>> listens to the probes and updates the statistics
>>>>>>>> • We process the statsProviders (annotations) for the
>>>>>>>> stats you have annotated to be exposed
>>>>>>>> • Exposing the stats through CLI (asadmin)
>>>>>>>> • Exposing the stats with REST
>>>>>>>> • GUI team is working on exposing these stats for some of
>>>>>>>> the critical modules
>>>>>>>> • Integrate with Gmbal to expose these stats as JMX/AMX
>>>>>>>> MBeans
>>>>>>>> • We also are ensuring backward compatibility for all the
>>>>>>>> V2 level stats
>>>>>>>> • Handling of config changes (from CLI,GUI or REST), so
>>>>>>>> the stats get enabled/disabled automatically
>>>>>>>> • We are using BTrace as our underlying infrastructure to do
>>>>>>>> the byte-code instrumentation of the provider classes so the
>>>>>>>> probes are fired and the registered listeners (StatsProviders)
>>>>>>>> are called.
>>>>>>>> Steps to see the monitoring in action
>>>>>>>>
>>>>>>>> You will need the BTrace agent running. You will need to start
>>>>>>>> the server with the following agent option (See Issue 8706, you
>>>>>>>> shouldn't need this step once its fixed.)
>>>>>>>>
>>>>>>>> $>java
>>>>>>>> -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=dumpClasses=true,unsafe=true
>>>>>>>> -jar glassfishv3/glassfish/modules/glassfish.jar
>>>>>>>>
>>>>>>>> To see the monitoring stats, you will have to turn on
>>>>>>>> monitoring for the interested modules (see the attached script
>>>>>>>> to turn them all to ON)
>>>>>>>>
>>>>>>>> There are various ways to see your stats,
>>>>>>>> • REST - http://localhost:4848/monitoring/domain
>>>>>>>> • asadmin - ./asadmin get --monitor=true "*"
>>>>>>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>>>>>>
>>>>>>>> Let us know if you run into any issues or if you have any
>>>>>>>> questions.
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> Prashanth
>>>>>>>>
>>>>>>>>
>>>>>>>> #!/bin/sh
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
>>>>>>>>
>>>>>>>> ./asadmin set
>>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>