users@glassfish.java.net

Re: Random 503 responses after server restart

From: Shing Wai Chan <shing.wai.chan_at_oracle.com>
Date: Tue, 01 Jun 2010 12:28:16 -0700

This may indicate the app is not completely started.
Do you do anything in servlet init() method?
Is the app ok if you wait for some time?
What exceptions do you get in the server.log?
Thanks.
      Shing Wai Chan

On 5/31/10 5:46 AM, glassfish_at_javadesktop.org wrote:
> Hello everyone,
>
> i have an issue, which is bugging me for some time now.
> I am running a recent glassfish 2.1.1 (just updated it to see if the problem is gone, which it is not) on a 64bit Debian Linux.
> The glassfish serves 6 instances of the same application using a different database and virtual host for each instance. The server is restarted each night for sanity reasons (i also experienced some slow downs in the past if i did not). The problem is, that sometimes for some reason not all applications start up properly, they just answer with a "503 - The requested service () is not currently available." message. I could not find anything usefull in the server logs. It is completely random if and which instance does not start up, sometimes even more than one. I noticed that a manual asadmin start/stop-domain always seemed to fix everything whereas doing it in a cron job needs several retries but this could also have been just luck.
> I wrote a small script run by cron which checks the server after each restart and re-restarts it until all applications respond properly. Ill attach its output from this morning at the bottom of this post. Interestingly, some applications/hosts crash more often than others but this, again, could be just coincidence.
>
> I'll also attach a logfile from one of the restarts to this post. There are some exceptions in it but they seem not problem-related to me.
>
> A current jvm report is also attached.
>
> This is the script's output (i replaced the domain names):
>
> www.host1.de gave a bad 503 respsonse!
> Restarting server...
> Domain sams stopped.
> Starting Domain sams, please wait.
> Default Log location is /home/glassfish/glassfish/domains/sams/logs/server.log.
> Redirecting output to /home/glassfish/glassfish/domains/sams/logs/server.log
> Domain sams is ready to receive client requests. Additional services are being started in background.
> Domain [sams] is running [Sun GlassFish Enterprise Server v2.1.1 ((v2.1 Patch06)(9.1_02 Patch12)) (build b31g-fcs)] with its configuration and logs at: [/home/glassfish/glassfish/domains].
> Admin Console is available at [http://localhost:8048].
> Use the same port [8048] for "asadmin" commands.
> User web applications are available at these URLs:
> [http://localhost:8080 https://localhost:8081 ].
> Following web-contexts are available:
> [/web1 /__wstx-services / / / / / ].
> Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
> [service:jmx:rmi:///jndi/rmi://sams-master:8086/jmxrmi] for domain management purposes.
> Domain listens on at least following ports for connections:
> [8080 8081 8048 8037 8038 8039 8086 ].
> Domain does not support application server clusters and other standalone instances.
>
> www.host2.de gave a bad 503 respsonse!
> www.host1.de gave a bad 503 respsonse!
> Restarting server...
> Domain sams stopped.
> Starting Domain sams, please wait.
> Default Log location is /home/glassfish/glassfish/domains/sams/logs/server.log.
> Redirecting output to /home/glassfish/glassfish/domains/sams/logs/server.log
> Domain sams is ready to receive client requests. Additional services are being started in background.
> Domain [sams] is running [Sun GlassFish Enterprise Server v2.1.1 ((v2.1 Patch06)(9.1_02 Patch12)) (build b31g-fcs)] with its configuration and logs at: [/home/glassfish/glassfish/domains].
> Admin Console is available at [http://localhost:8048].
> Use the same port [8048] for "asadmin" commands.
> User web applications are available at these URLs:
> [http://localhost:8080 https://localhost:8081 ].
> Following web-contexts are available:
> [/web1 /__wstx-services / / / / / ].
> Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
> [service:jmx:rmi:///jndi/rmi://sams-master:8086/jmxrmi] for domain management purposes.
> Domain listens on at least following ports for connections:
> [8080 8081 8048 8037 8038 8039 8086 ].
> Domain does not support application server clusters and other standalone instances.
>
> www.host2.de gave a bad 503 respsonse!
> Restarting server...
> Domain sams stopped.
> Starting Domain sams, please wait.
> Default Log location is /home/glassfish/glassfish/domains/sams/logs/server.log.
> Redirecting output to /home/glassfish/glassfish/domains/sams/logs/server.log
> Domain sams is ready to receive client requests. Additional services are being started in background.
> Domain [sams] is running [Sun GlassFish Enterprise Server v2.1.1 ((v2.1 Patch06)(9.1_02 Patch12)) (build b31g-fcs)] with its configuration and logs at: [/home/glassfish/glassfish/domains].
> Admin Console is available at [http://localhost:8048].
> Use the same port [8048] for "asadmin" commands.
> User web applications are available at these URLs:
> [http://localhost:8080 https://localhost:8081 ].
> Following web-contexts are available:
> [/web1 /__wstx-services / / / / / ].
> Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
> [service:jmx:rmi:///jndi/rmi://sams-master:8086/jmxrmi] for domain management purposes.
> Domain listens on at least following ports for connections:
> [8080 8081 8048 8037 8038 8039 8086 ].
> Domain does not support application server clusters and other standalone instances.
>
> I'd appreciate any help with this!
>
> Thanks in advance, Robert
> [Message sent by forum member 'roben_rb']
>
> http://forums.java.net/jive/thread.jspa?messageID=472121
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>