A server reboot seems to have fixed whatever was the problem. Thanks for your help Tom!
----- Original Message -----
From: modjklist_at_comcast.net
To: "Tom Mueller" <tom.mueller_at_oracle.com>
Cc: users_at_glassfish.java.net
Sent: Tuesday, July 3, 2012 1:39:37 PM
Subject: Re: using asadmin to stop domain running on a specific port number
I can SSH into the server using X11 and go through the directories fine. The server.log file is very large (tens of thousands of lines long) and it opens, edits, and saves almost instantly if I were to delete a line, for example). I use Eclipse remotely to saves from/to the server, and it is still fast. Nothing seems unusual other than GlassFish stop/start/restart times. I'm using a web application deployed on GlassFish and it also runs fine.
It's just a simple dedicated (not virtual) Linux server without SAN.
----- Original Message -----
From: "Tom Mueller" <tom.mueller_at_oracle.com>
To: users_at_glassfish.java.net
Cc: modjklist_at_comcast.net
Sent: Tuesday, July 3, 2012 1:33:42 PM
Subject: Re: using asadmin to stop domain running on a specific port number
It looks like there is something generally wrong with the system, maybe
with file access. Is the rest of the system slow? How long does a find
on the filesystem take? Are you running on a virtualized system? If so,
is there a problem with other guests on the system? Is there a SAN? Is
it behaving normally?
Tom
On 7/3/2012 3:24 PM, modjklist_at_comcast.net wrote:
> Also, server.log is showing very long times and variations for loading processes. For example, before today this was typical:
>
> [#|2012-06-17T12:55:15.547-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2 (23) startup time : Felix (1,297ms), startup services(4,717ms), total(6,014ms)|#]
>
> [#|2012-06-17T12:55:18.608-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=17;_ThreadName=Thread-2;|CORE10010: Loading application __admingui done in 2,559 ms|#]
>
> [#|2012-06-17T12:55:14.667-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application myapp_web done in 1,385 ms|#]
>
>
> But now I'm seeing:
>
> [#|2012-07-03T10:58:45.829-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2 (23) startup time : Felix (558,616ms), startup services(130,568ms), total(689,184ms)|#]
>
> [#|2012-07-03T10:44:06.096-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=18;_ThreadName=Thread-2;|CORE10010: Loading application __admingui done in 487,963 ms|#]
>
> [#|2012-07-03T10:35:02.525-0700|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application myapp_web done in 622,519 ms|#]
>
> The above shows what used to take 5 seconds now takes 1800 seconds! Any idea how to fix this?
>
>
> ----- Original Message -----
> From: modjklist_at_comcast.net
> To: users_at_glassfish.java.net
> Cc: "Tom Mueller"<tom.mueller_at_oracle.com>
> Sent: Tuesday, July 3, 2012 1:03:57 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> Some more information... eventually, after about 30 minutes, I get the time-out message below. Here's the sequence I typed in the linux command line:
>
> kill -9 20037
> asadmin> start-domain domain1
> Waiting for domain1 to start .
> Successfully started the domain : domain1
> domain Location: ...(I removed)...
> Log File: ...(I removed)...
> Admin Port: 4850
> Command start-domain executed successfully.
> asadmin> restart-domain --port 4850
> Timed out waiting for the server to restart
> Command restart-domain failed.
>
> Any idea what is going on?
>
> ----- Original Message -----
> From: modjklist_at_comcast.net
> To: "Tom Mueller"<tom.mueller_at_oracle.com>
> Cc: users_at_glassfish.java.net
> Sent: Tuesday, July 3, 2012 12:39:01 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> Thanks Tom,
>
> I'm using GlassFish version 3.1.2.
>
> When I type:
>
> jps -v | grep 20037
>
> I see the directory of my GlassFish install, as you mention below.
>
> Thanks for connecting the "domain1" with the --port 4850 option. I understand now it's need for stopping (and restarting) the domain.
>
> I'm not aware about the bug you refer to. So, I tried issuing:
>
> kill -9 20037
>
> then I typed:
>
> asadmin>start-domain domain1
>
> and it started successfully. Then I tried this:
>
> asadmin>restart-domain --port 4850
>
> and nothing gets returned (it just sits there; normally it takes 2-3 seconds to complete). This is what I observed earlier today, and may be related to a process hanging somewhere. The last line in the server.log file is "The Admin Console application is loaded", which occurred 5 minutes after entering the restart command above. But the linux command line for asadmin> appears hung, although I can control-C to get a command prompt again. Any idea what is going on?
>
>
>
> ----- Original Message -----
> From: "Tom Mueller"<tom.mueller_at_oracle.com>
> To: modjklist_at_comcast.net
> Cc: users_at_glassfish.java.net
> Sent: Tuesday, July 3, 2012 12:14:45 PM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> What version of GlassFish are you running?
>
> To see what process 20037 is, you can do:
>
> jps -v | grep 20037
>
> This will show must of the Java command line that was used to start the process. Assuming it is a GlassFish process, you'll see an argument on that command line that looks like this: -Djavax.net.ssl.keyStore=/scratch/trm/test/glassfish3/glassfish/domains/domain1/config/keystore.jks
> This will show what domain directory the process is running from. If this is the directory of your GlassFish install, then the list-domains output is either wrong due to a bug (that's why I asked about the version) or the process is hung in some way that it is not responding to the list-domains query for it's status.
>
> If the process is hung, you might want to try a kill -9 to stop it.
>
> The --port option is not needed for the start-domain command. The port number is only needed by restart-domain to stop the process (it tries a graceful shutdown by sending it a command to shut down).
>
> WRT your question about the connection between domain1 and port 4850, the "domain1" argument to start-domain is the directory name within the domain directory. In that directory, start-domain looks for the config/domain.xml file, and in that file, it finds the port number (4850) to use when starting the process.
>
> Tom
>
> On 7/3/2012 1:57 PM, modjklist_at_comcast.net wrote:
>
>
> (including list)
>
> Thanks Tom, Here's what I get when I type what you suggested:
>
> [root_at_host9 ~]# netstat -tulpn | grep 4850
> tcp 0 0 :::4850 :::* LISTEN 20037/java
>
> Not sure what it means. I know that I setup the GlassFish web administrators console to use port 4850 during installation (instead of default port 4848). So that should be what is using port 4850.
>
> I'm confused though. For whatever reason, I've been using the following to start/stop/restart domain1 (showing restart for example):
>
> adasmin> restart-domain --port 4850
>
> and it has always worked fine. Today when I try it I get:
>
> asadmin> start-domain --port 4850
> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
> Command start-domain failed.
> asadmin> start-domain domain1 --port 4850
> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
> Command start-domain failed.
>
> How would I start domain1?
> What's the connection between domain1 and port 4850?
>
>
>
> ----- Original Message -----
> From: "Tom Mueller"<tom.mueller_at_oracle.com>
> To: users_at_glassfish.java.net
> Cc: modjklist_at_comcast.net
> Sent: Tuesday, July 3, 2012 11:44:58 AM
> Subject: Re: using asadmin to stop domain running on a specific port number
>
> The list-domains command will only identify domains in the domain
> directory that it is operating on. By default, this is the
> "glassfish/domains" directory in the install directory for the asadmin
> command. However, there is a --domaindir option for list-domains that
> allows specifying something else.
>
> The process that you have running on port 4850 could be a GlassFish
> server process from another domain that is in a non-default domains
> directory, or it could be from a domain from another install of
> GlassFish on the same host, or the process could be some completely
> different process that happens to be using port 4850.
>
> On Linux, you can use "netstat -tulpn" to find out what process is
> listening on a port, i.e.:
>
> $ netstat -tulpn | grep 4850
>
> Tom
>
>
> On 7/3/2012 1:21 PM, modjklist_at_comcast.net wrote:
>> I am observing the following:
>>
>> asadmin> list-domains
>> domain1 not running
>> Command list-domains executed successfully.
>> asadmin> start-domain domain1 --port 4850
>> There is a process already using the admin port 4850 -- it probably is another instance of a GlassFish server.
>> Command start-domain failed.
>>
>> Shouldn't the "list-domains" show running domains on port 4850? Or, more importantly, how do I stop whatever is running on port 4850, so I can start domain1 on that port?