dev@glassfish.java.net

Re: Update on Monitoring in V3

From: Prashanth Abbagani <Prashanth.Abbagani_at_Sun.COM>
Date: Thu, 16 Jul 2009 13:00:01 -0700

Alexis Moussine-Pouchkine wrote:
>
> On Jul 16, 2009, at 19:12, Prashanth Abbagani wrote:
>
>> Alexis Moussine-Pouchkine wrote:
>>> thanks Prashanth, this is useful.
>>>
>>> Note that if monitoring is not enabled, the requests fail with an
>>> NPE (promoted build 55):
>> Doesn't seem to happen in the monitoring infra. Can I see your
>> provider class, is this checked in?
>
> Sorry I should have been more detailed.
> Simply start b55 (monitoring OFF) and hit
> http://localhost:4848/monitoring/domain/server/jvm/runtime
> It should trigger the exception below.
> -Alexis
I will give it a try, but this is happening in rest, the underlying
problem could be Monitoring Infra. Rajeshwar, can you pls debug the
problem and let me know the root cause here.

thanks
Prashanth
>
>>>
>>>> SEVERE: service exception
>>>> java.lang.NullPointerException
>>>> at
>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.xmlForPrimitiveValue(TreeNodeXmlProvider.java:168)
>>>>
>>>> at
>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getAttributes(TreeNodeXmlProvider.java:118)
>>>>
>>>> at
>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.getXml(TreeNodeXmlProvider.java:95)
>>>>
>>>> at
>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:87)
>>>>
>>>> at
>>>> org.glassfish.admin.rest.provider.TreeNodeXmlProvider.writeTo(TreeNodeXmlProvider.java:61)
>>>>
>>>> at
>>>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:254)
>>>>
>>>> at
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:744)
>>>>
>>>> at
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:667)
>>>>
>>>> at
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:658)
>>>>
>>>> at
>>>> com.sun.jersey.server.impl.container.grizzly.GrizzlyContainer.service(GrizzlyContainer.java:132)
>>>>
>>>> at
>>>> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:165)
>>>>
>>>> at
>>>> com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:100)
>>>>
>>>> at
>>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:208)
>>>>
>>>> at
>>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
>>>>
>>>> at
>>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
>>>> at
>>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
>>>> at
>>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
>>>>
>>>> at
>>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
>>>>
>>>> at
>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
>>>>
>>>> at
>>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
>>>>
>>>> at
>>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
>>>>
>>>> at
>>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>
>>>> at
>>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
>>>>
>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>> at java.lang.Thread.run(Thread.java:637)
>>>
>>> An error message suggesting to turn on monitoring would be nice.
>> I agree, I need to understand at what point is this happening. If
>> this is in the ProbeProvider, the monitoring flag has nothing to do
>> with this.
>>>
>>> Also, are core classes monitoring loaded prior to any call?
>>> I see several messages at startup (even without the javaagent
>>> option) which seem to have an impact on startup time.
>> The Providers are discovered by default during the module's startup.
>> The retrasformation of ProbeProvider (which is the only
>> compute-intensive operation) will only happen when there are
>> listeners (StatsProviders) registered and these statsProviders are
>> registered when the corresponding monitoring flag is turned on. So
>> there is no overhead when the monitoring is turned off. We do have
>> the Monitoring startup class which keeps discovering the providers as
>> they come (even without the javaagent), this is a required step and I
>> dont think it effects the startup time significantly.
>>>
>>> thanks,
>>> -Alexis
>>>
>>> On Jul 16, 2009, at 24:52, Prashanth Abbagani wrote:
>>>
>>>> Hello,
>>>>
>>>> The monitoring in V3 is now integrated and working with several
>>>> stats from various module showing up. See the instructions towards
>>>> the end to which explain on how to view these stats.
>>>>
>>>> Following are the features of monitoring that were accomplished
>>>> before the soft-code freeze deadline.
>>>> • Monitoring Probe Providers bundled by the module owners are
>>>> discovered automatically and processed so probes are being fired
>>>> when it visits module's code path
>>>> • Following is done with the monitoring StatsProviders which
>>>> listens to the probes and updates the statistics
>>>> • We process the statsProviders (annotations) for the stats
>>>> you have annotated to be exposed
>>>> • Exposing the stats through CLI (asadmin)
>>>> • Exposing the stats with REST
>>>> • GUI team is working on exposing these stats for some of
>>>> the critical modules
>>>> • Integrate with Gmbal to expose these stats as JMX/AMX MBeans
>>>> • We also are ensuring backward compatibility for all the V2
>>>> level stats
>>>> • Handling of config changes (from CLI,GUI or REST), so the
>>>> stats get enabled/disabled automatically
>>>> • We are using BTrace as our underlying infrastructure to do the
>>>> byte-code instrumentation of the provider classes so the probes are
>>>> fired and the registered listeners (StatsProviders) are called.
>>>> Steps to see the monitoring in action
>>>>
>>>> You will need the BTrace agent running. You will need to start the
>>>> server with the following agent option (See Issue 8706, you
>>>> shouldn't need this step once its fixed.)
>>>>
>>>> $>java
>>>> -javaagent:/space/GFV3_BLD/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=dumpClasses=true,unsafe=true
>>>> -jar glassfishv3/glassfish/modules/glassfish.jar
>>>>
>>>> To see the monitoring stats, you will have to turn on monitoring
>>>> for the interested modules (see the attached script to turn them
>>>> all to ON)
>>>>
>>>> There are various ways to see your stats,
>>>> • REST - http://localhost:4848/monitoring/domain
>>>> • asadmin - ./asadmin get --monitor=true "*"
>>>> • JMX/AMX - Use JConsole to connect to 'localhost:8686'
>>>>
>>>> Let us know if you run into any issues or if you have any questions.
>>>>
>>>> thanks
>>>> Prashanth
>>>>
>>>>
>>>> #!/bin/sh
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
>>>>
>>>> ./asadmin set
>>>> configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>