users@glassfish.java.net

Re: How to enable JMX monitoring

From: Jan Algermissen <jan.algermissen_at_nordsc.com>
Date: Thu, 4 Oct 2012 12:04:58 +0200

Tim,


On Oct 3, 2012, at 11:12 PM, Tim Quinn wrote:

> Hi, again, Jan.
>
> I have just installed a very recent build of GlassFish 4 and I can successfully connect to it using jconsole and jvisualvm remotely (after changing the admin password, enabling secure admin, and restarting the domain).
>
> A couple of possibilities come to mind.

Having tried a few more times with GF3 also, I am ready to give up trying.

One last possibility would be that there is a firewall in place that I am not aware of that happens to block non standard, higher port numbers.

To get that possibility out of the way, I'd try one last thing and use a fixed port.

Again, for this (fixing the otherwise dynamic JMX port for Glassfish), I found various 'this seems to work' postings. Do you know how it should be done or what a best practice would be?


Jan



>
> First, the version of Java should be recent on both ends. In response to Tom's note I saw that you have a very recent Java installation on the server. What Java version is on your client?
>
> Second, if as you conjectured, the cert containing localhost is part of the problem, you can try this:
>
> asadmin stop-domain
> asadmin delete-domain domain1
> asadmin create-domain domain1 #(explicitly enter "admin" for the username and respond to the password prompt with a non-empty password)
> asadmin start-domain
> asadmin enable-secure-admin
> asadmin restart-domain
>
> Then try again to connect to the DAS remotely using jvisualvm.
>
> Creating the domain again, among other things, creates self-signed certs that have the DAS's actual host name built in (instead of localhost). If that was the problem then recreating the domain might help.
>
> - Tim
>
> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>
>>
>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>
>>>> Hi Tim,
>>>>
>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>
>>>>> Hello, Jan.
>>>>>
>>>>> Please take a quick look at this recent blog entry:
>>>>>
>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>
>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>
>>>>
>>>> Thanks very much. Much cleaner than anything else so far.
>>>>
>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>
>>>
>>> Update: the certificate of GF seems to be the problem.
>>
>>
>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>
>> Question is now: how do I get a valid certificate installed. Help welcome.
>>
>> Meanwhile I plough through the blogs on that topic.
>>
>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>
>> Thanks for all the help so far!
>>
>>
>> Jan
>>
>>
>>
>>
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Stacktrace below, in case you have a vague idea.
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>> Trace:
>>>>
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>> java.net.ConnectException: Operation timed out
>>>>
>>>> [ and more ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>
>>>>> - Tim
>>>>>
>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>
>>>>>> Hence the question:
>>>>>>
>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>
>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>
>>>>
>>>
>>
>
> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>
>>
>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>
>>>> Hi Tim,
>>>>
>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>
>>>>> Hello, Jan.
>>>>>
>>>>> Please take a quick look at this recent blog entry:
>>>>>
>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>
>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>
>>>>
>>>> Thanks very much. Much cleaner than anything else so far.
>>>>
>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>
>>>
>>> Update: the certificate of GF seems to be the problem.
>>
>>
>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>
>> Question is now: how do I get a valid certificate installed. Help welcome.
>>
>> Meanwhile I plough through the blogs on that topic.
>>
>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>
>> Thanks for all the help so far!
>>
>>
>> Jan
>>
>>
>>
>>
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Stacktrace below, in case you have a vague idea.
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>> Trace:
>>>>
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>> java.net.ConnectException: Operation timed out
>>>>
>>>> [ and more ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>
>>>>> - Tim
>>>>>
>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>
>>>>>> Hence the question:
>>>>>>
>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>
>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>
>>>>
>>>
>>
>