dev@glassfish.java.net

Re: glassfish hang (blocking all upgrades)

From: Bobby Bissett <Robert.Bissett_at_Sun.COM>
Date: Mon, 09 Nov 2009 13:52:03 -0500

More info. Nothing else than what I sent before is going to the log or std out/err, but if I hack the upgrade tool to log everything coming out of the java process I get the output below (which should be in the log unless that's what's hanging). The "OUT" and "ERR" strings are the names passed to my stream-watching classes:

ERR:Nov 9, 2009 1:46:27 PM com.sun.common.util.logging.LoggingConfigImpl openPropFile
ERR:INFO: Cannot read logging.properties file.
ERR:Nov 9, 2009 1:46:27 PM com.sun.enterprise.admin.launcher.GFLauncherLogger info
ERR:INFO: JVM invocation command line:
ERR:/kauai/beach/jdk1.6.0_17/bin/java
ERR:-cp
ERR:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/modules/glassfish.jar
ERR:-XX:MaxPermSize=192m
ERR:-XX:NewRatio=2
ERR:-Xmx512m
ERR:-client
ERR:-javaagent:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true
ERR:-Dsun.rmi.dgc.client.gcInterval=3600000
ERR:-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
ERR:-Djavax.net.ssl.keyStore=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/keystore.jks
ERR:-Djava.security.policy=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/server.policy
ERR:-Dsun.rmi.dgc.server.gcInterval=3600000
ERR:-Dcom.sun.aas.instanceRoot=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1
ERR:-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
ERR:-Djava.endorsed.dirs=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/lib/endorsed
ERR:-Dcom.sun.aas.installRoot=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish
ERR:-Djava.ext.dirs=/kauai/beach/jdk1.6.0_17/lib/ext:/kauai/beach/jdk1.6.0_17/jre/lib/ext:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/lib/ext:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/javadb/lib
ERR:-Djavax.net.ssl.trustStore=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/cacerts.jks
ERR:-Dcom.sun.enterprise.taglisteners=jsf-impl.jar
ERR:-Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
ERR:-Djava.security.auth.login.config=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/login.conf
ERR:-Djava.library.path=/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/lib:/kauai/beach/jdk1.6.0_17/jre/lib/sparc/server:/kauai/beach/jdk1.6.0_17/jre/lib/sparc:/kauai/beach/jdk1.6.0_17/lib/sparc:/usr/jdk/packages/lib/sparc:/usr/lib
ERR:com.sun.enterprise.glassfish.bootstrap.ASMain
ERR:-domainname
ERR:domain1
ERR:-asadmin-args
ERR:start-domain,,,--upgrade,,,--domaindir,,,/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/,,,domain1
ERR:-instancename
ERR:server
ERR:-verbose
ERR:true
ERR:-debug
ERR:false
ERR:-asadmin-classpath
ERR:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/modules/admin-cli.jar
ERR:-asadmin-classname
ERR:com.sun.enterprise.admin.cli.AsadminMain
ERR:-upgrade
ERR:true
ERR:-domaindir
ERR:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1
ERR:-read-stdin
ERR:true
ERR:Nov 9, 2009 1:46:27 PM com.sun.enterprise.admin.launcher.GFLauncherLogger info
ERR:INFO: Successfully launched in 144 msec.
ERR:Nov 9, 2009 1:46:28 PM com.sun.enterprise.glassfish.bootstrap.ASMain main
ERR:INFO: Launching GlassFish on Felix platform
OUT:
OUT:Welcome to Felix
OUT:================
OUT:
ERR:Nov 9, 2009 1:46:40 PM org.glassfish.config.support.DomainXml$DomainXmlReader parse2ndTime
ERR:INFO: Forced to parse file:/kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/domain.xml twice because we didn't see <server> before <config>
ERR:Nov 9, 2009 1:46:41 PM org.hibernate.validation.util.Version <clinit>
ERR:INFO: Hibernate Validator null
ERR:Nov 9, 2009 1:46:41 PM org.hibernate.validation.engine.resolver.DefaultTraversableResolver detectJPA
ERR:INFO: Instantiated an instance of org.hibernate.validation.engine.resolver.JPATraversableResolver.
ERR:Nov 9, 2009 1:46:43 PM org.glassfish.config.support.GrizzlyConfigSchemaMigrator postConstruct
ERR:SEVERE: Failure while upgrading domain.xml. Please redeploy
ERR:org.jvnet.hk2.config.TransactionFailure: Constraints for this bean violated.
ERR: Message = minThreadPoolSize must be greater than or equal to 2
ERR: at org.jvnet.hk2.config.WriteableView.canCommit(WriteableView.java:287)
ERR: at org.jvnet.hk2.config.Transaction.commit(Transaction.java:96)
ERR: at org.jvnet.hk2.config.ConfigSupport._apply(ConfigSupport.java:173)
ERR: at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:130)
ERR: at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:109)
ERR: at org.glassfish.config.support.GrizzlyConfigSchemaMigrator.rectifyThreadPools(GrizzlyConfigSchemaMigrator.java:122)
ERR: at org.glassfish.config.support.GrizzlyConfigSchemaMigrator.postConstruct(GrizzlyConfigSchemaMigrator.java:59)
ERR: at com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:174)
ERR: at com.sun.hk2.component.ConstructorWomb$1.run(ConstructorWomb.java:87)
ERR: at java.security.AccessController.doPrivileged(Native Method)
ERR: at com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:84)
ERR: at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:77)
ERR: at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
ERR: at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
ERR: at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
ERR: at org.jvnet.hk2.component.Habitat$1.get(Habitat.java:276)
ERR: at java.util.AbstractList$Itr.next(AbstractList.java:345)
ERR: at org.glassfish.config.support.DomainXml.upgrade(DomainXml.java:128)
ERR: at org.glassfish.config.support.DomainXml.run(DomainXml.java:108)
ERR: at com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.populateConfig(AbstractModulesRegistryImpl.java:159)
ERR: at com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.createHabitat(AbstractModulesRegistryImpl.java:144)
ERR: at com.sun.enterprise.module.bootstrap.Main.createHabitat(Main.java:418)
ERR: at org.jvnet.hk2.osgiadapter.HK2Main.start(HK2Main.java:122)
ERR: at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:640)
ERR: at org.apache.felix.framework.Felix.activateBundle(Felix.java:1700)
ERR: at org.apache.felix.framework.Felix.startBundle(Felix.java:1622)
ERR: at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:915)
ERR: at org.jvnet.hk2.osgimain.Main.start(Main.java:140)
ERR: at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:640)
OUT:{felix.fileinstall.poll (ms) = 2000, felix.fileinstall.dir = /kauai/beach/ws/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/./load, felix.fileinstall.debug = -1, felix.fileinstall.bundles.new.start = true, felix.fileinstall.tmpdir = /var/tmp/fileinstall--8885348265315461560, felix.fileinstall.filter = null}
ERR: at org.apache.felix.framework.Felix.activateBundle(Felix.java:1700)
ERR: at org.apache.felix.framework.Felix.startBundle(Felix.java:1622)
ERR: at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1077)
ERR: at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
ERR: at java.lang.Thread.run(Thread.java:619)
ERR:Nov 9, 2009 1:46:43 PM Main start
ERR:WARNING: Exception while starting bundle com.sun.enterprise.osgi-adapter [214]
ERR:org.osgi.framework.BundleException: Activator start error in bundle com.sun.enterprise.osgi-adapter [214].
ERR: at org.apache.felix.framework.Felix.activateBundle(Felix.java:1751)
ERR: at org.apache.felix.framework.Felix.startBundle(Felix.java:1622)
ERR: at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:915)
ERR: at org.jvnet.hk2.osgimain.Main.start(Main.java:140)
ERR: at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:640)
ERR: at org.apache.felix.framework.Felix.activateBundle(Felix.java:1700)
ERR: at org.apache.felix.framework.Felix.startBundle(Felix.java:1622)
ERR: at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1077)
ERR: at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:264)
ERR: at java.lang.Thread.run(Thread.java:619)
ERR:Caused by: java.lang.RuntimeException: org.jvnet.hk2.config.TransactionFailure: Constraints for this bean violated.
ERR: Message = minThreadPoolSize must be greater than or equal to 2
ERR: at org.glassfish.config.support.GrizzlyConfigSchemaMigrator.postConstruct(GrizzlyConfigSchemaMigrator.java:66)
ERR: at com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:174)
ERR: at com.sun.hk2.component.ConstructorWomb$1.run(ConstructorWomb.java:87)
ERR: at java.security.AccessController.doPrivileged(Native Method)
ERR: at com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:84)
ERR: at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:77)
ERR: at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:58)
ERR: at com.sun.hk2.component.LazyInhabitant.get(LazyInhabitant.java:107)
ERR: at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:60)
ERR: at org.jvnet.hk2.component.Habitat$1.get(Habitat.java:276)
ERR: at java.util.AbstractList$Itr.next(AbstractList.java:345)
ERR: at org.glassfish.config.support.DomainXml.upgrade(DomainXml.java:128)
ERR: at org.glassfish.config.support.DomainXml.run(DomainXml.java:108)
ERR: at com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.populateConfig(AbstractModulesRegistryImpl.java:159)
ERR: at com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.createHabitat(AbstractModulesRegistryImpl.java:144)
ERR: at com.sun.enterprise.module.bootstrap.Main.createHabitat(Main.java:418)
ERR: at org.jvnet.hk2.osgiadapter.HK2Main.start(HK2Main.java:122)
ERR: at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:640)
ERR: at org.apache.felix.framework.Felix.activateBundle(Felix.java:1700)
ERR: ... 9 more
ERR:Caused by: org.jvnet.hk2.config.TransactionFailure: Constraints for this bean violated.
ERR: Message = minThreadPoolSize must be greater than or equal to 2
ERR: at org.jvnet.hk2.config.WriteableView.canCommit(WriteableView.java:287)
ERR: at org.jvnet.hk2.config.Transaction.commit(Transaction.java:96)
ERR: at org.jvnet.hk2.config.ConfigSupport._apply(ConfigSupport.java:173)
ERR: at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:130)
ERR: at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:109)
ERR: at org.glassfish.config.support.GrizzlyConfigSchemaMigrator.rectifyThreadPools(GrizzlyConfigSchemaMigrator.java:122)
ERR: at org.glassfish.config.support.GrizzlyConfigSchemaMigrator.postConstruct(GrizzlyConfigSchemaMigrator.java:59)
ERR: ... 27 more