One last follow up on this. Thanks to Nigel for solving this.
The root cause of this problem is a configuration setting that comes
with the IT-hosted Linux VMs. These VMs have an /etc/sysctl.conf file
with the following entry:
net.ipv4.ip_local_port_range = 1024 65000
This is under a comment containing "COMMON SECTION to OTN
recommendations and FST env" so apparently Oracle recommends making this
change to a Linux configuration. The effect of this change is that the
ephemeral range for TCP ports is changed to 1024-65000 rather than the
IANA recommended range of 49152-65535. This means that even something
like logging into a host via telnet or ssh can cause one of the ports
that your GlassFish install is planning to use to be allocated for your
client session, meaning that your server will not start successfully.
Beware.
I'll try filing a ticket on this.
Tom
On 9/10/2010 9:53 AM, Tom Mueller wrote:
> I've been looking at the ports that cluster members listen on, and I'm
> seeing some mysterious ports. Can someone please identify what these
> ports are being used for? Also, how are the port numbers allocated?
>
> Here are the ports used by a typical instance, running without GMS
> enabled and with JMS in embedded mode:
>
> TCP *:28081 (LISTEN) < HTTP_LISTENER_PORT
> TCP *:28182 (LISTEN) < HTTP_SSL_LISTENER_PORT
> TCP *:24849 (LISTEN) < ASADMIN_LISTENER_PORT
> TCP 127.0.0.1:26667 (LISTEN) < OSGI_SHELL_TELNET_PORT
> TCP *:27677 (LISTEN) < JMS_PROVIDER_PORT
> TCP *:27777 (LISTEN) < ?????
> TCP *:16426 (LISTEN) < ?????
> TCP *:52978 (LISTEN) < ?????
> TCP *:32874 (LISTEN) < ?????
> TCP *:4910 (LISTEN) < ?????
> TCP *:28687 (LISTEN) < JMX_SYSTEM_CONNECTOR_PORT
>
> The concern that I have with these ports is that they appear to be
> dynamically allocated, because they change each time the instance is
> run and for different instances, but except for one, they are
> allocated from the outside the region for allocating dynamic ports
> (which is 49152 to 65535). And if the port number chosen happens to be
> the one that another instance on the same system is going to use, then
> that instance will fail to start because it can't bind to it's port.
>
> I also curious as to what determines the address that these sockets
> are bound too, for the multihomed server use case.
>
> Thanks.
> Tom
>
>
>