As far as I can tell so far, the 503 error page is actually being served by SJSWS 7 which is sitting in front of my glassfish cluster as the load balancer. Checking the logs there, I see that glassfish2 (my problem machine) is reported as unhealthy. I think this is the reason for the 503: when I kill glassfish1 and the load balancer only has glassfish2 to send requests to, it throws the 503 error page instead because glassfish2 is reported as unhealthy.
So, now to why glassfish2 is unhealthy. I discovered two things:
1) I use Sun ONE Application Server cache tags in my JSPs, but had forgotten to add JVM options to allow the app to access those tags.
2) I use Hibernate Search and had forgotten to create the directory where my search indexes were supposed to be stored.
For the first problem above, here's the log that indicated the problem:
Log Level: SEVERE
Logger: com.sun.org.apache.commons.modeler.Registry
Name-Value Pairs: _ThreadID=11;_ThreadName=pool-1-thread-2;_RequestID=9a83947b-bd79-4c08-a73d-f068a8efc3d8;
Record Number: 5610
Message ID: Null component com.sun.appserv
Complete Message:
type=JspMonitor,name=Talk,WebModule=//server/,J2EEApplication=null,J2EEServer=g2-i1
It had been a while since I'd set up my dev machine for the appserv tags, so I'd forgotten to make the changes to the production server. I found this by going to the JSP where the error occurred to try and figure out what happened and that's when I saw the appserv taglib import at the top of the page, which reminded me that those had some dependencies.
For the second problem above, the log message that put me on the right track was:
Log Level: SEVERE
Logger: javax.enterprise.system.container.web
Name-Value Pairs: _ThreadID=12;_ThreadName=pool-1-thread-4;_RequestID=00c14e56-ed0a-4ab0-8ca1-e1960f3a83d0;
Record Number: 4378
Message ID: WEB0207
Complete Message:
Error starting web context StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[] on virtual server server java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Unable to initialize: com.stag.beans.Pool at com.sun.enterprise.web.WebModuleListener.loadPersistenceUnits(WebModuleListener.java:193) at com.sun.enterprise.web.WebModuleListener.lifecycleEvent(WebModuleListener.java:168) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:143) at org.apache.catalina.core.StandardContext.start(StandardContext.java:5086) at com.sun.enterprise.web.WebModule.start(WebModule.java:327) at com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:58) at com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBase.java:296) at com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.java:330) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Unable to initialize: com.stag.beans.Pool at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:737) at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:127) at com.sun.enterprise.server.PersistenceUnitLoaderImpl.load(PersistenceUnitLoaderImpl.java:149) at com.sun.enterprise.server.PersistenceUnitLoaderImpl.load(PersistenceUnitLoaderImpl.java:84) at com.sun.enterprise.web.WebModuleListener.loadPersistenceUnits(WebModuleListener.java:190) ... 13 more Caused by: org.hibernate.HibernateException: Unable to initialize: com.stag.beans.Pool at org.hibernate.search.store.DirectoryProviderFactory.createDirectoryProvider(DirectoryProviderFactory.java:147) at org.hibernate.search.store.DirectoryProviderFactory.createDirectoryProviders(DirectoryProviderFactory.java:75) at org.hibernate.search.impl.SearchFactoryImpl.initDocumentBuilders(SearchFactoryImpl.java:260) at org.hibernate.search.impl.SearchFactoryImpl.<init>(SearchFactoryImpl.java:94) at org.hibernate.search.impl.SearchFactoryImpl.getSearchFactory(SearchFactoryImpl.java:172) at org.hibernate.search.event.FullTextIndexEventListener.initialize(FullTextIndexEventListener.java:44) at org.hibernate.event.EventListeners.initializeListeners(EventListeners.java:356) at org.hibernate.cfg.Configuration.getInitializedEventListeners(Configuration.java:1304) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1294) at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:915) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:730) ... 17 more Caused by: [b]org.hibernate.search.SearchException: Cannot write into index directory: /var/lucene/indexes[/b] at org.hibernate.search.util.DirectoryProviderHelper.determineIndexDir(DirectoryProviderHelper.java:86) at org.hibernate.search.store.FSDirectoryProvider.initialize(FSDirectoryProvider.java:28) at org.hibernate.search.store.DirectoryProviderFactory.createDirectoryProvider(DirectoryProviderFactory.java:144) ... 27 more
The key part being the bold part above.
Why do I post this? Because it's a good example of why a node in your cluster might not start up. Namely, it might not have anything to do with the cluster configuration, and finding out why an instance isn't working involves some digging in the logs.
[Message sent by forum member 'rwillie6' (rwillie6)]
http://forums.java.net/jive/thread.jspa?messageID=245710