dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Alexis Moussine-Pouchkine <alexis.mp_at_sun.com>
Date: Thu, 16 Jul 2009 23:00:51 +0200

On Jul 16, 2009, at 22:51, Rajeshwar Patil wrote:

> Prashanth Abbagani wrote:
>> Alexis Moussine-Pouchkine wrote:
>>>
>>> On Jul 16, 2009, at 19:12, Prashanth Abbagani wrote:
>>>
>>>> Alexis Moussine-Pouchkine wrote:
>>>>> thanks Prashanth, this is useful.
>>>>>
>>>>> Note that if monitoring is not enabled, the requests fail with
>>>>> an NPE (promoted build 55):
>>>> Doesn't seem to happen in the monitoring infra. Can I see your
>>>> provider class, is this checked in?
>>>
>>> Sorry I should have been more detailed.
>>> Simply start b55 (monitoring OFF) and hit http://localhost:4848/monitoring/domain/server/jvm/runtime
>>> It should trigger the exception below.
>>> -Alexis
>> I will give it a try, but this is happening in rest, the underlying
>> problem could be Monitoring Infra. Rajeshwar, can you pls debug the
>> problem and let me know the root cause here.
> Hello Prashanth,
> Monitoring infrastructure is returning attribute list (nodes with
> null value) even when monitoring if OFF. Regardless, I will also
> put a null check in REST code.

Hi Rajeshwar,

Is there anyway that situation could be translated into a message to
the user saying that he should turn monitoring to HIGH (or LOW) if
he's trying to monitoring GF using the REST interface?

thanks!
-Alexis

>
> Thanks
> Rajeshwar
>>
>> thanks
>> Prashanth
>>>
>>>>>
>>>>>> SEVERE: service exception
>>>>>> java.lang.NullPointerException
>>>>>> at
>>>>>> org
>>>>>> .glassfish
>>>>>> .admin
>>>>>> .rest
>>>>>> .provider
>>>>>> .TreeNodeXmlProvider
>>>>>> .xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>>>> at
>>>>>> org
>>>>>> .glassfish
>>>>>> .admin
>>>>>> .rest
>>>>>> .provider
>>>>>> .TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>>>> at
>>>>>> org
>>>>>> .glassfish
>>>>>> .admin
>>>>>> .rest
>>>>>> .provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>>>> at
>>>>>> org
>>>>>> .glassfish
>>>>>> .admin
>>>>>> .rest
>>>>>> .provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:
>>>>>> 87)
>>>>>> at
>>>>>> org
>>>>>> .glassfish
>>>>>> .admin
>>>>>> .rest
>>>>>> .provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:
>>>>>> 61)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .jersey
>>>>>> .spi.container.ContainerResponse.write(ContainerResponse.java:
>>>>>> 254)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .jersey
>>>>>> .server
>>>>>> .impl
>>>>>> .application
>>>>>> .WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .jersey
>>>>>> .server
>>>>>> .impl
>>>>>> .application
>>>>>> .WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .jersey
>>>>>> .server
>>>>>> .impl
>>>>>> .application
>>>>>> .WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .jersey
>>>>>> .server
>>>>>> .impl
>>>>>> .container
>>>>>> .grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
>>>>>> 165)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
>>>>>> 100)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .enterprise
>>>>>> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:
>>>>>> 208)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>>>> at
>>>>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:
>>>>>> 655)
>>>>>> at
>>>>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:
>>>>>> 905)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:
>>>>>> 161)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .DefaultProtocolChain
>>>>>> .executeProtocolFilter(DefaultProtocolChain.java:136)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 103)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 89)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:
>>>>>> 76)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:
>>>>>> 53)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:637)
>>>>>
>>>>> An error message suggesting to turn on monitoring would be nice.
>>>> I agree, I need to understand at what point is this happening. If
>>>> this is in the ProbeProvider, the monitoring flag has nothing to
>>>> do with this.
>>>>>
>>>>> Also, are core classes monitoring loaded prior to any call?
>>>>> I see several messages at startup (even without the javaagent
>>>>> option) which seem to have an impact on startup time.
>>>> The Providers are discovered by default during the module's
>>>> startup. The retrasformation of ProbeProvider (which is the only
>>>> compute-intensive operation) will only happen when there are
>>>> listeners (StatsProviders) registered and these statsProviders
>>>> are registered when the corresponding monitoring flag is turned
>>>> on. So there is no overhead when the monitoring is turned off. We
>>>> do have the Monitoring startup class which keeps discovering the
>>>> providers as they come (even without the javaagent), this is a
>>>> required step and I dont think it effects the startup time
>>>> significantly.
>>>>>
>>>>> thanks,
>>>>> -Alexis
>>>>>
>>>>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> The monitoring in V3 is now integrated and working with several
>>>>>> stats from various module showing up. See the instructions
>>>>>> towards the end to which explain on how to view these stats.
>>>>>>
>>>>>> Following are the features of monitoring that were accomplished
>>>>>> before the soft-code freeze deadline.
>>>>>> • Monitoring Probe Providers bundled by the module owners are
>>>>>> discovered automatically and processed so probes are being
>>>>>> fired when it visits module's code path
>>>>>> • Following is done with the monitoring StatsProviders which
>>>>>> listens to the probes and updates the statistics
>>>>>> • We process the statsProviders (annotations) for the
>>>>>> stats you have annotated to be exposed
>>>>>> • Exposing the stats through CLI (asadmin)
>>>>>> • Exposing the stats with REST
>>>>>> • GUI team is working on exposing these stats for some of
>>>>>> the critical modules
>>>>>> • Integrate with Gmbal to expose these stats as JMX/AMX
>>>>>> MBeans
>>>>>> • We also are ensuring backward compatibility for all the
>>>>>> V2 level stats
>>>>>> • Handling of config changes (from CLI,GUI or REST), so
>>>>>> the stats get enabled/disabled automatically
>>>>>> • We are using BTrace as our underlying infrastructure to do
>>>>>> the byte-code instrumentation of the provider classes so the
>>>>>> probes are fired and the registered listeners (StatsProviders)
>>>>>> are called.
>>>>>> Steps to see the monitoring in action
>>>>>>
>>>>>> You will need the BTrace agent running. You will need to start
>>>>>> the server with the following agent option (See Issue 8706, you
>>>>>> shouldn't need this step once its fixed.)
>>>>>>
>>>>>> $>java -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/
>>>>>> monitor/btrace-agent.jar=dumpClasses=true,unsafe=true -jar
>>>>>> glassfishv3/glassfish/modules/glassfish.jar
>>>>>>
>>>>>> To see the monitoring stats, you will have to turn on
>>>>>> monitoring for the interested modules (see the attached script
>>>>>> to turn them all to ON)
>>>>>>
>>>>>> There are various ways to see your stats,
>>>>>> • REST - http://localhost:4848/monitoring/domain
>>>>>> • asadmin - ./asadmin get --monitor=true "*"
>>>>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>>>>
>>>>>> Let us know if you run into any issues or if you have any
>>>>>> questions.
>>>>>>
>>>>>> thanks
>>>>>> Prashanth
>>>>>>
>>>>>>
>>>>>> #!/bin/sh
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.connector-connection-pool=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.connector-service=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.ejb-container=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.http-service=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.jms-service=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.jvm=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.orb=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.thread-pool=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.transaction-service=HIGH
>>>>>> ./asadmin set configs.config.server-config.monitoring-
>>>>>> service.module-monitoring-levels.web-container=HIGH
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>