Ok I sorted this out. What I missed about this:
<jvm-options>-Dcom.sun.aas.jconsole.server.cbport=18687</jvm-options>
Is server is the default instance name and I'm running in a cluster so this works better:
-Dcom.sun.aas.jconsole.${com.sun.aas.instanceName}.cbport=18687
The other thing to change was the service url, it now looks like this:
service:jmx:rmi://hostname.foo.com:18687/jndi/rmi://hostname.foo.com:18686/jmxrmi
With the 2 ports open throught the firewall I have JMX working.
[Message sent by forum member 'dbsquared' (dbortnichak_at_scrippsnetworks.com)]
http://forums.java.net/jive/thread.jspa?messageID=378868