users@glassfish.java.net

Re: How to enable JMX monitoring

From: Jan Algermissen <jan.algermissen_at_nordsc.com>
Date: Wed, 3 Oct 2012 23:34:05 +0200

Hi Tim,

On Oct 3, 2012, at 11:12 PM, Tim Quinn wrote:

> Hi, again, Jan.
>
> I have just installed a very recent build of GlassFish 4 and I can successfully connect to it using jconsole and jvisualvm remotely (after changing the admin password, enabling secure admin, and restarting the domain).
>
> A couple of possibilities come to mind.
>
> First, the version of Java should be recent on both ends. In response to Tom's note I saw that you have a very recent Java installation on the server. What Java version is on your client?

Same as server's.

>
> Second, if as you conjectured, the cert containing localhost is part of the problem, you can try this:
>
> asadmin stop-domain
> asadmin delete-domain domain1
> asadmin create-domain domain1 #(explicitly enter "admin" for the username and respond to the password prompt with a non-empty password)
> asadmin start-domain
> asadmin enable-secure-admin
> asadmin restart-domain
>

Thanks - that is a great trick indeed. I have spend the last hours to dig through the keytool stuff with no real success in trying what 'brute force' does in seconds :-)

Very nice.

However...


> Then try again to connect to the DAS remotely using jvisualvm.

VisualVM still does not do it and e.g. Safari still chockes on the cert saying it has not been validated by a third party. Strange .. isn't the whole idea of self signing that no 3rd party is needed?

Strange that it works for you and not for me. I am on a fresh, hosted root server so there should not be any trouble, actually.

Thanks anyhow. I appreciate it very much!

Jan

>
> Creating the domain again, among other things, creates self-signed certs that have the DAS's actual host name built in (instead of localhost). If that was the problem then recreating the domain might help.
>
> - Tim
>
> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>
>>
>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>
>>>> Hi Tim,
>>>>
>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>
>>>>> Hello, Jan.
>>>>>
>>>>> Please take a quick look at this recent blog entry:
>>>>>
>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>
>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>
>>>>
>>>> Thanks very much. Much cleaner than anything else so far.
>>>>
>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>
>>>
>>> Update: the certificate of GF seems to be the problem.
>>
>>
>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>
>> Question is now: how do I get a valid certificate installed. Help welcome.
>>
>> Meanwhile I plough through the blogs on that topic.
>>
>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>
>> Thanks for all the help so far!
>>
>>
>> Jan
>>
>>
>>
>>
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Stacktrace below, in case you have a vague idea.
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>> Trace:
>>>>
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>> java.net.ConnectException: Operation timed out
>>>>
>>>> [ and more ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>
>>>>> - Tim
>>>>>
>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>
>>>>>> Hence the question:
>>>>>>
>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>
>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>
>>>>
>>>
>>
>
> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>
>>
>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>
>>>> Hi Tim,
>>>>
>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>
>>>>> Hello, Jan.
>>>>>
>>>>> Please take a quick look at this recent blog entry:
>>>>>
>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>
>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>
>>>>
>>>> Thanks very much. Much cleaner than anything else so far.
>>>>
>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>
>>>
>>> Update: the certificate of GF seems to be the problem.
>>
>>
>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>
>> Question is now: how do I get a valid certificate installed. Help welcome.
>>
>> Meanwhile I plough through the blogs on that topic.
>>
>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>
>> Thanks for all the help so far!
>>
>>
>> Jan
>>
>>
>>
>>
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Stacktrace below, in case you have a vague idea.
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>> Trace:
>>>>
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>> java.net.ConnectException: Operation timed out
>>>>
>>>> [ and more ]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>
>>>>> - Tim
>>>>>
>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>
>>>>>> Hence the question:
>>>>>>
>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>
>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>
>>>>
>>>
>>
>