It looks like there is something generally wrong with the system, maybe
with file access. Is the rest of the system slow? How long does a find
on the filesystem take? Are you running on a virtualized system? If so,
is there a problem with other guests on the system? Is there a SAN? Is
it behaving normally?
Tom
On 7/3/2012 3:24 PM, modjklist_at_comcast.net wrote:
> Also, server.log is showing very long times and variations for loading processes. For example, before today this was typical:
>
> [#|2012-06-17T12:55:15.547-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2 (23) startup time : Felix (1,297ms), startup services(4,717ms), total(6,014ms)|#]
>
> [#|2012-06-17T12:55:18.608-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=17;_ThreadName=Thread-2;|CORE10010: Loading application __admingui done in 2,559 ms|#]
>
> [#|2012-06-17T12:55:14.667-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application myapp_web done in 1,385 ms|#]
>
>
> But now I'm seeing:
>
> [#|2012-07-03T10:58:45.829-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2 (23) startup time : Felix (558,616ms), startup services(130,568ms), total(689,184ms)|#]
>
> [#|2012-07-03T10:44:06.096-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=18;_ThreadName=Thread-2;|CORE10010: Loading application __admingui done in 487,963 ms|#]
>
> [#|2012-07-03T10:35:02.525-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application myapp_web done in 622,519 ms|#]
>
> The above shows what used to take 5 seconds now takes 1800 seconds! Any idea how to fix this?
>
>
> ----- Original Message -----
> From: modjklist_at_comcast.net
> To: users_at_glassfish.java.net
> Cc: "Tom Mueller"<tom.mueller_at_oracle.com>
> Sent: Tuesday, July 3, 2012 1:03:57 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> Some more information... eventually, after about 30 minutes, I get the time-out message below. Here's the sequence I typed in the linux command line:
>
> kill -9 20037
> asadmin> start-domain domain1
> Waiting for domain1 to start .
> Successfully started the domain : domain1
> domain Location: ...(I removed)...
> Log File: ...(I removed)...
> Admin Port: 4850
> Command start-domain executed successfully.
> asadmin> restart-domain --port 4850
> Timed out waiting for the server to restart
> Command restart-domain failed.
>
> Any idea what is going on?
>
> ----- Original Message -----
> From: modjklist_at_comcast.net
> To: "Tom Mueller"<tom.mueller_at_oracle.com>
> Cc: users_at_glassfish.java.net
> Sent: Tuesday, July 3, 2012 12:39:01 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> Thanks Tom,
>
> I'm using GlassFish version 3.1.2.
>
> When I type:
>
> jps -v | grep 20037
>
> I see the directory of my GlassFish install, as you mention below.
>
> Thanks for connecting the "domain1" with the --port 4850 option. I understand now it's need for stopping (and restarting) the domain.
>
> I'm not aware about the bug you refer to. So, I tried issuing:
>
> kill -9 20037
>
> then I typed:
>
> asadmin>start-domain domain1
>
> and it started successfully. Then I tried this:
>
> asadmin>restart-domain --port 4850
>
> and nothing gets returned (it just sits there; normally it takes 2-3 seconds to complete). This is what I observed earlier today, and may be related to a process hanging somewhere. The last line in the server.log file is "The Admin Console application is loaded", which occurred 5 minutes after entering the restart command above. But the linux command line for asadmin> appears hung, although I can control-C to get a command prompt again. Any idea what is going on?
>
>
>
> ----- Original Message -----
> From: "Tom Mueller"<tom.mueller_at_oracle.com>
> To: modjklist_at_comcast.net
> Cc: users_at_glassfish.java.net
> Sent: Tuesday, July 3, 2012 12:14:45 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> What version of GlassFish are you running?
>
> To see what process 20037 is, you can do:
>
> jps -v | grep 20037
>
> This will show must of the Java command line that was used to start the process. Assuming it is a GlassFish process, you'll see an argument on that command line that looks like this: -Djavax.net.ssl.keyStore=/scratch/trm/test/glassfish3/glassfish/domains/domain1/config/keystore.jks
> This will show what domain directory the process is running from. If this is the directory of your GlassFish install, then the list-domains output is either wrong due to a bug (that's why I asked about the version) or the process is hung in some way that it is not responding to the list-domains query for it's status.
>
> If the process is hung, you might want to try a kill -9 to stop it.
>
> The --port option is not needed for the start-domain command. The port number is only needed by restart-domain to stop the process (it tries a graceful shutdown by sending it a command to shut down).
>
> WRT your question about the connection between domain1 and port 4850, the "domain1" argument to start-domain is the directory name within the domain directory. In that directory, start-domain looks for the config/domain.xml file, and in that file, it finds the port number (4850) to use when starting the process.
>
> Tom
>
> On 7/3/2012 1:57 PM, modjklist_at_comcast.net wrote:
>
>
> (including list)
>
> Thanks Tom, Here's what I get when I type what you suggested:
>
> [root_at_host9 ~]# netstat -tulpn | grep 4850
> tcp 0 0 :::4850 :::* LISTEN 20037/java
>
> Not sure what it means. I know that I setup the GlassFish web administrators console to use port 4850 during installation (instead of default port 4848). So that should be what is using port 4850.
>
> I'm confused though. For whatever reason, I've been using the following to start/stop/restart domain1 (showing restart for example):
>
> adasmin> restart-domain --port 4850
>
> and it has always worked fine. Today when I try it I get:
>
> asadmin> start-domain --port 4850
> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
> Command start-domain failed.
> asadmin> start-domain domain1 --port 4850
> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
> Command start-domain failed.
>
> How would I start domain1?
> What's the connection between domain1 and port 4850?
>
>
>
> ----- Original Message -----
> From: "Tom Mueller"<tom.mueller_at_oracle.com>
> To: users_at_glassfish.java.net
> Cc: modjklist_at_comcast.net
> Sent: Tuesday, July 3, 2012 11:44:58 AM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> The list-domains command will only identify domains in the domain
> directory that it is operating on. By default, this is the
> "glassfish/domains" directory in the install directory for the asadmin
> command. However, there is a --domaindir option for list-domains that
> allows specifying something else.
>
> The process that you have running on port 4850 could be a GlassFish
> server process from another domain that is in a non-default domains
> directory, or it could be from a domain from another install of
> GlassFish on the same host, or the process could be some completely
> different process that happens to be using port 4850.
>
> On Linux, you can use "netstat -tulpn" to find out what process is
> listening on a port, i.e.:
>
> $ netstat -tulpn | grep 4850
>
> Tom
>
>
> On 7/3/2012 1:21 PM, modjklist_at_comcast.net wrote:
>> I am observing the following:
>>
>> asadmin> list-domains
>> domain1 not running
>> Command list-domains executed successfully.
>> asadmin> start-domain domain1 --port 4850
>> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
>> Command start-domain failed.
>>
>> Shouldn't the "list-domains" show running domains on port 4850? Or, more importantly, how do I stop whatever is running on port 4850, so I can start domain1 on that port?