users@glassfish.java.net

Re: How to enable JMX monitoring

From: Tim Quinn <tim.quinn_at_oracle.com>
Date: Thu, 4 Oct 2012 09:26:42 -0500

Hello, Jan.

You mentioned earlier in the thread that you see no errors logged in the DAS's server.log. I would think that if the client's connection attempt was reaching the server and then things failed that there would be some evidence of that in the server.log. Given that the server.log contains no errors about this, your idea about the firewall blocking requests might be on the right track.

You can change the JMX port using

        asadmin set server-config.admin-service.jmx-connector.system.port=xxxx
        asadmin restart-domain

Replace xxxx with the port you want to use.

Also some firewalls can be set to log blocked accesses. If yours does then that could give some more information about this.

- Tim

On Oct 4, 2012, at 5:04 AM, Jan Algermissen wrote:

> Tim,
>
>
> On Oct 3, 2012, at 11:12 PM, Tim Quinn wrote:
>
>> Hi, again, Jan.
>>
>> I have just installed a very recent build of GlassFish 4 and I can successfully connect to it using jconsole and jvisualvm remotely (after changing the admin password, enabling secure admin, and restarting the domain).
>>
>> A couple of possibilities come to mind.
>
> Having tried a few more times with GF3 also, I am ready to give up trying.
>
> One last possibility would be that there is a firewall in place that I am not aware of that happens to block non standard, higher port numbers.
>
> To get that possibility out of the way, I'd try one last thing and use a fixed port.
>
> Again, for this (fixing the otherwise dynamic JMX port for Glassfish), I found various 'this seems to work' postings. Do you know how it should be done or what a best practice would be?
>
>
> Jan
>
>
>
>>
>> First, the version of Java should be recent on both ends. In response to Tom's note I saw that you have a very recent Java installation on the server. What Java version is on your client?
>>
>> Second, if as you conjectured, the cert containing localhost is part of the problem, you can try this:
>>
>> asadmin stop-domain
>> asadmin delete-domain domain1
>> asadmin create-domain domain1 #(explicitly enter "admin" for the username and respond to the password prompt with a non-empty password)
>> asadmin start-domain
>> asadmin enable-secure-admin
>> asadmin restart-domain
>>
>> Then try again to connect to the DAS remotely using jvisualvm.
>>
>> Creating the domain again, among other things, creates self-signed certs that have the DAS's actual host name built in (instead of localhost). If that was the problem then recreating the domain might help.
>>
>> - Tim
>>
>> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>>
>>>>
>>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>>
>>>>>> Hello, Jan.
>>>>>>
>>>>>> Please take a quick look at this recent blog entry:
>>>>>>
>>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>>
>>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>>
>>>>>
>>>>> Thanks very much. Much cleaner than anything else so far.
>>>>>
>>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>>
>>>>
>>>> Update: the certificate of GF seems to be the problem.
>>>
>>>
>>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>>
>>> Question is now: how do I get a valid certificate installed. Help welcome.
>>>
>>> Meanwhile I plough through the blogs on that topic.
>>>
>>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>>
>>> Thanks for all the help so far!
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Stacktrace below, in case you have a vague idea.
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>>>>
>>>>> Trace:
>>>>>
>>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>>> java.net.ConnectException: Operation timed out
>>>>>
>>>>> [ and more ]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>>
>>>>>> - Tim
>>>>>>
>>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>>
>>>>>>> Hence the question:
>>>>>>>
>>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>>
>>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> On Oct 3, 2012, at 3:20 PM, Jan Algermissen wrote:
>>
>>>
>>> On Oct 3, 2012, at 9:39 PM, Jan Algermissen wrote:
>>>
>>>>
>>>> On Oct 3, 2012, at 9:31 PM, Jan Algermissen wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> On Oct 3, 2012, at 7:09 PM, Tim Quinn wrote:
>>>>>
>>>>>> Hello, Jan.
>>>>>>
>>>>>> Please take a quick look at this recent blog entry:
>>>>>>
>>>>>> https://blogs.oracle.com/quinn/entry/jmx_monitoring_of_glassfish_servers
>>>>>>
>>>>>> This does not count as "official documentation" but the steps there do work.
>>>>>>
>>>>>
>>>>> Thanks very much. Much cleaner than anything else so far.
>>>>>
>>>>> But no luck for me. The visualvm logs suggest some SSL related problems.
>>>>
>>>>
>>>> Update: the certificate of GF seems to be the problem.
>>>
>>>
>>> I guess the problem is that the cert is for localhost and that does not match the hostname I point my visualvm to.
>>>
>>> Question is now: how do I get a valid certificate installed. Help welcome.
>>>
>>> Meanwhile I plough through the blogs on that topic.
>>>
>>> (What makes me wonder is that what I am trying to do, namely: monitor a GF remotely with VisualVM, seems like a standard use case. But I feel like the only person having problems there. Strange)
>>>
>>> Thanks for all the help so far!
>>>
>>>
>>> Jan
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> Jan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Stacktrace below, in case you have a vague idea.
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>>>>
>>>>> Trace:
>>>>>
>>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi://xxxxxxxxxx:8686/jndi/rmi://xxxxxxxxxxxx:8686/jmxrmi)
>>>>> java.io.EOFException: SSL peer shut down incorrectly
>>>>> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>>>>> Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
>>>>> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:654)
>>>>> at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
>>>>> Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>>>>> Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
>>>>> javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
>>>>> at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>>>>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>>>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>>>>> at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
>>>>> Caused: java.io.IOException: Failed to retrieve RMIServer stub
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:496)
>>>>> [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:440)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:220)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:209)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
>>>>> at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
>>>>> at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:279)
>>>>> at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:183)
>>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:293)
>>>>> at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:270)
>>>>> at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
>>>>> at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
>>>>> at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)
>>>>> INFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(3574)
>>>>> java.net.ConnectException: Operation timed out
>>>>>
>>>>> [ and more ]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> The formatting of some of the images there leaves a little to be desired, but I think it's clear enough what's going on.
>>>>>>
>>>>>> - Tim
>>>>>>
>>>>>> On Oct 3, 2012, at 11:38 AM, Jan Algermissen wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> despite the large number of Q&As on the subject out there, I feel pretty lost. Most of them talk about how people eventually, somehow managed to get remote JMX monitoring to work. I just cannot find an authoritative answer, how it *should* be done.
>>>>>>>
>>>>>>> Hence the question:
>>>>>>>
>>>>>>> Given a fresh install of Glassfish 4, what are the things I must do to enable remote monitoring via VisualVM? Is there any official documentation on the subject?
>>>>>>>
>>>>>>> (I am not going through a firewall, so there should not be an issue with the second port VisualVM opens up).
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>