dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Rajeshwar Patil <Rajeshwar.Patil_at_Sun.COM>
Date: Thu, 16 Jul 2009 13:51:10 -0700

Prashanth Abbagani wrote:
> Alexis Moussine-Pouchkine wrote:
>>
>> On Jul 16, 2009, at 19:12, Prashanth Abbagani wrote:
>>
>>> Alexis Moussine-Pouchkine wrote:
>>>> thanks Prashanth, this is useful.
>>>>
>>>> Note that if monitoring is not enabled, the requests fail with an
>>>> NPE (promoted build 55):
>>> Doesn't seem to happen in the monitoring infra. Can I see your
>>> provider class, is this checked in?
>>
>> Sorry I should have been more detailed.
>> Simply start b55 (monitoring OFF) and hit
>> http://localhost:4848/monitoring/domain/server/jvm/runtime
>> It should trigger the exception below.
>> -Alexis
> I will give it a try, but this is happening in rest, the underlying
> problem could be Monitoring Infra. Rajeshwar, can you pls debug the
> problem and let me know the root cause here.
Hello Prashanth,
Monitoring infrastructure is returning attribute list (nodes with null
value) even when monitoring if OFF. Regardless, I will also put a null
check in REST code.

Thanks
Rajeshwar
>
> thanks
> Prashanth
>>
>>>>
>>>>> SEVERE: service exception
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>>>
>>>>> at
>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>>>
>>>>> at
>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>>>
>>>>> at
>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:87)
>>>>>
>>>>> at
>>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:61)
>>>>>
>>>>> at
>>>>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:254)
>>>>>
>>>>> at
>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>>>
>>>>> at
>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>>>
>>>>> at
>>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>>>
>>>>> at
>>>>> com.sun.jersey.server.impl.container.grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:165)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:208)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
>>>>> at
>>>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
>>>>> at
>>>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>>
>>>>> at
>>>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>>>
>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:637)
>>>>
>>>> An error message suggesting to turn on monitoring would be nice.
>>> I agree, I need to understand at what point is this happening. If
>>> this is in the ProbeProvider, the monitoring flag has nothing to do
>>> with this.
>>>>
>>>> Also, are core classes monitoring loaded prior to any call?
>>>> I see several messages at startup (even without the javaagent
>>>> option) which seem to have an impact on startup time.
>>> The Providers are discovered by default during the module's startup.
>>> The retrasformation of ProbeProvider (which is the only
>>> compute-intensive operation) will only happen when there are
>>> listeners (StatsProviders) registered and these statsProviders are
>>> registered when the corresponding monitoring flag is turned on. So
>>> there is no overhead when the monitoring is turned off. We do have
>>> the Monitoring startup class which keeps discovering the providers
>>> as they come (even without the javaagent), this is a required step
>>> and I dont think it effects the startup time significantly.
>>>>
>>>> thanks,
>>>> -Alexis
>>>>
>>>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> The monitoring in V3 is now integrated and working with several
>>>>> stats from various module showing up. See the instructions towards
>>>>> the end to which explain on how to view these stats.
>>>>>
>>>>> Following are the features of monitoring that were accomplished
>>>>> before the soft-code freeze deadline.
>>>>> • Monitoring Probe Providers bundled by the module owners are
>>>>> discovered automatically and processed so probes are being fired
>>>>> when it visits module's code path
>>>>> • Following is done with the monitoring StatsProviders which
>>>>> listens to the probes and updates the statistics
>>>>> • We process the statsProviders (annotations) for the stats
>>>>> you have annotated to be exposed
>>>>> • Exposing the stats through CLI (asadmin)
>>>>> • Exposing the stats with REST
>>>>> • GUI team is working on exposing these stats for some of
>>>>> the critical modules
>>>>> • Integrate with Gmbal to expose these stats as JMX/AMX MBeans
>>>>> • We also are ensuring backward compatibility for all the
>>>>> V2 level stats
>>>>> • Handling of config changes (from CLI,GUI or REST), so the
>>>>> stats get enabled/disabled automatically
>>>>> • We are using BTrace as our underlying infrastructure to do
>>>>> the byte-code instrumentation of the provider classes so the
>>>>> probes are fired and the registered listeners (StatsProviders) are
>>>>> called.
>>>>> Steps to see the monitoring in action
>>>>>
>>>>> You will need the BTrace agent running. You will need to start the
>>>>> server with the following agent option (See Issue 8706, you
>>>>> shouldn't need this step once its fixed.)
>>>>>
>>>>> $>java
>>>>> -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=dumpClasses=true,unsafe=true
>>>>> -jar glassfishv3/glassfish/modules/glassfish.jar
>>>>>
>>>>> To see the monitoring stats, you will have to turn on monitoring
>>>>> for the interested modules (see the attached script to turn them
>>>>> all to ON)
>>>>>
>>>>> There are various ways to see your stats,
>>>>> • REST - http://localhost:4848/monitoring/domain
>>>>> • asadmin - ./asadmin get --monitor=true "*"
>>>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>>>
>>>>> Let us know if you run into any issues or if you have any questions.
>>>>>
>>>>> thanks
>>>>> Prashanth
>>>>>
>>>>>
>>>>> #!/bin/sh
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
>>>>>
>>>>> ./asadmin set
>>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>