dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Rajeshwar Patil <Rajeshwar.Patil_at_Sun.COM>
Date: Thu, 16 Jul 2009 18:43:21 -0700

Alexis Moussine-Pouchkine wrote:
>
> On Jul 16, 2009, at 22:51, Rajeshwar Patil wrote:
>
>> Prashanth Abbagani wrote:
>>> Alexis Moussine-Pouchkine wrote:
>>>>
>>>> On Jul 16, 2009, at 19:12, Prashanth Abbagani wrote:
>>>>
>>>>> Alexis Moussine-Pouchkine wrote:
>>>>>> thanks Prashanth, this is useful.
>>>>>>
>>>>>> Note that if monitoring is not enabled, the requests fail with an
>>>>>> NPE (promoted build 55):
>>>>> Doesn't seem to happen in the monitoring infra. Can I see your
>>>>> provider class, is this checked in?
>>>>
>>>> Sorry I should have been more detailed.
>>>> Simply start b55 (monitoring OFF) and hit
>>>> http://localhost:4848/monitoring/domain/server/jvm/runtime
>>>> It should trigger the exception below.
>>>> -Alexis
>>> I will give it a try, but this is happening in rest, the underlying
>>> problem could be Monitoring Infra. Rajeshwar, can you pls debug the
>>> problem and let me know the root cause here.
>> Hello Prashanth,
>> Monitoring infrastructure is returning attribute list (nodes with
>> null value) even when monitoring if OFF. Regardless, I will also put
>> a null check in REST code.
>
> Hi Rajeshwar,
>
> Is there anyway that situation could be translated into a message to
> the user saying that he should turn monitoring to HIGH (or LOW) if
> he's trying to monitoring GF using the REST interface?
Hello Alexis,
I believe user should never see top level JVM node itself if monitoring
is OFF for JVM.
We can not do provide message in a generic way. We can do this for known
or default modules but it won't be possible for new/third party modules
which can bring in new monitoring nodes with their own configuration
flags. Being said this, I can see whether I can provide massage for the
top level resource (http://localhost:port/management/domain) when no
child node is available at all. This is the most likely use case user
will run into.

Thanks
Rajeshwar
>
> thanks!
> -Alexis
>
>>
>> Thanks
>> Rajeshwar
>>>
>>> thanks
>>> Prashanth
>>>>
>>>>>>
>>>>>>> SEVERE: service exception
>>>>>>> java.lang.NullPointerException
>>>>>>> at
>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>>>>>
>>>>>>> at
>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>>>>>
>>>>>>> at
>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>>>>>
>>>>>>> at
>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:87)
>>>>>>>
>>>>>>> at
>>>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:61)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:254)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.jersey.server.impl.container.grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:165)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:208)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
>>>>>>> at
>>>>>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>>>>
>>>>>>> at
>>>>>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>>>>>
>>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:637)
>>>>>>
>>>>>> An error message suggesting to turn on monitoring would be nice.
>>>>> I agree, I need to understand at what point is this happening. If
>>>>> this is in the ProbeProvider, the monitoring flag has nothing to
>>>>> do with this.
>>>>>>
>>>>>> Also, are core classes monitoring loaded prior to any call?
>>>>>> I see several messages at startup (even without the javaagent
>>>>>> option) which seem to have an impact on startup time.
>>>>> The Providers are discovered by default during the module's
>>>>> startup. The retrasformation of ProbeProvider (which is the only
>>>>> compute-intensive operation) will only happen when there are
>>>>> listeners (StatsProviders) registered and these statsProviders are
>>>>> registered when the corresponding monitoring flag is turned on. So
>>>>> there is no overhead when the monitoring is turned off. We do have
>>>>> the Monitoring startup class which keeps discovering the providers
>>>>> as they come (even without the javaagent), this is a required step
>>>>> and I dont think it effects the startup time significantly.
>>>>>>
>>>>>> thanks,
>>>>>> -Alexis
>>>>>>
>>>>>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> The monitoring in V3 is now integrated and working with several
>>>>>>> stats from various module showing up. See the instructions
>>>>>>> towards the end to which explain on how to view these stats.
>>>>>>>
>>>>>>> Following are the features of monitoring that were accomplished
>>>>>>> before the soft-code freeze deadline.
>>>>>>> • Monitoring Probe Providers bundled by the module owners are
>>>>>>> discovered automatically and processed so probes are being fired
>>>>>>> when it visits module's code path
>>>>>>> • Following is done with the monitoring StatsProviders which
>>>>>>> listens to the probes and updates the statistics
>>>>>>> • We process the statsProviders (annotations) for the
>>>>>>> stats you have annotated to be exposed
>>>>>>> • Exposing the stats through CLI (asadmin)
>>>>>>> • Exposing the stats with REST
>>>>>>> • GUI team is working on exposing these stats for some of
>>>>>>> the critical modules
>>>>>>> • Integrate with Gmbal to expose these stats as JMX/AMX
>>>>>>> MBeans
>>>>>>> • We also are ensuring backward compatibility for all the
>>>>>>> V2 level stats
>>>>>>> • Handling of config changes (from CLI,GUI or REST), so
>>>>>>> the stats get enabled/disabled automatically
>>>>>>> • We are using BTrace as our underlying infrastructure to do
>>>>>>> the byte-code instrumentation of the provider classes so the
>>>>>>> probes are fired and the registered listeners (StatsProviders)
>>>>>>> are called.
>>>>>>> Steps to see the monitoring in action
>>>>>>>
>>>>>>> You will need the BTrace agent running. You will need to start
>>>>>>> the server with the following agent option (See Issue 8706, you
>>>>>>> shouldn't need this step once its fixed.)
>>>>>>>
>>>>>>> $>java
>>>>>>> -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=dumpClasses=true,unsafe=true
>>>>>>> -jar glassfishv3/glassfish/modules/glassfish.jar
>>>>>>>
>>>>>>> To see the monitoring stats, you will have to turn on monitoring
>>>>>>> for the interested modules (see the attached script to turn them
>>>>>>> all to ON)
>>>>>>>
>>>>>>> There are various ways to see your stats,
>>>>>>> • REST - http://localhost:4848/monitoring/domain
>>>>>>> • asadmin - ./asadmin get --monitor=true "*"
>>>>>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>>>>>
>>>>>>> Let us know if you run into any issues or if you have any
>>>>>>> questions.
>>>>>>>
>>>>>>> thanks
>>>>>>> Prashanth
>>>>>>>
>>>>>>>
>>>>>>> #!/bin/sh
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
>>>>>>>
>>>>>>> ./asadmin set
>>>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>