dev@glassfish.java.net

Re: why does Glassfish V3 not respond in 90 seconds during startup

From: Amy Roh <Amelia.Roh_at_Sun.COM>
Date: Wed, 29 Apr 2009 13:11:00 -0700

I used to see the deadlock issue intermittently since mac doesn't have
jdk6 u11. Killing the process and restarting the server usually works
but I cannot start the server today at all after updating the workspace.

This seems to be a different issue (at least for me) from the logging
deadlock issue.

I have deleted /.m2/repository/ and did "mvn -U clean install".

bash-3.2$ glassfish/bin/asadmin start-domain

Domain (domain1) did not respond in 90 seconds. It means it is still
coming up
or it has failed to come up. Check server.log for details.

Staring server with --verbose gives me - (full log below)

com.sun.enterprise.module.ResolveError: Failed to start
org.glassfish.common.glassfish-mbeanserver(Glassfish MBeanServer
support):3.0.0.SNAPSHOT

...

Caused by: org.osgi.framework.BundleException: Unresolved constraint in
bundle 37: package; (package=org.glassfish.api.branding)

How do I resolve this?

Thanks,
Amy


=================

bash-3.2$ glassfish/bin/asadmin start-domain --verbose

Apr 29, 2009 12:55:59 PM
com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: JVM invocation command line:
/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java
-cp
/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
-XX:MaxPermSize=192m
-XX:NewRatio=2
-XX:+LogVMOutput
-XX:LogFile=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/logs/jvm.log
-Xmx512m
-client
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djavax.net.ssl.trustStore=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/cacerts.jks
-Djavax.net.ssl.keyStore=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/keystore.jks
-Djava.security.policy=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/server.policy
-Dcom.sun.aas.instanceRoot=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Djava.security.auth.login.config=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/config/login.conf
-Djava.endorsed.dirs=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/lib/endorsed
-Dcom.sun.aas.installRoot=/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish
-Djava.ext.dirs=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/lib/ext:/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/jre/lib/ext:/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1/lib/ext:/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/javadb/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-domainname
domain1
-asadmin-args
start-domain,,,--verbose
-instancename
server
-verbose
true
-debug
false
-asadmin-classpath
glassfish/bin/../modules/admin-cli.jar
-asadmin-classname
com.sun.enterprise.admin.cli.AsadminMain
-domaindir
/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/domains/domain1
Apr 29, 2009 12:55:59 PM
com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: Successfully launched in 15 msec.
Apr 29, 2009 12:55:59 PM com.sun.enterprise.glassfish.bootstrap.ASMain main
INFO: Launching GlassFish on Apache Felix OSGi platform

Welcome to Felix.
=================

felix.fileinstall.poll (ms) 5000
felix.fileinstall.dir
/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/modules
felix.fileinstall.debug 1
felix.fileinstall.bundles.new.start false
ERROR: Error starting
file:/Users/Amy/v3-4-20/v3/distributions/glassfish/target/glassfishv3/glassfish/modules/osgi-main.jar
(org.osgi.framework.BundleException: Activator start error in bundle
com.sun.enterprise.osgi-adapter [164].)
com.sun.enterprise.module.ResolveError: Failed to start
org.glassfish.common.glassfish-mbeanserver(Glassfish MBeanServer
support):3.0.0.SNAPSHOT
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:143)
        at
org.jvnet.hk2.osgiadapter.OSGiModuleImpl$1$1$1.loadClass(OSGiModuleImpl.java:280)
        at com.sun.hk2.component.LazyInhabitant.fetch(LazyInhabitant.java:91)
        at com.sun.hk2.component.LazyInhabitant.type(LazyInhabitant.java:78)
        at org.jvnet.hk2.component.Habitat.addIndex(Habitat.java:149)
        at com.sun.hk2.component.InhabitantsParser.add(InhabitantsParser.java:154)
        at
com.sun.hk2.component.InhabitantsParser.parse(InhabitantsParser.java:133)
        at
org.jvnet.hk2.osgiadapter.OSGiModuleImpl.parseInhabitants(OSGiModuleImpl.java:312)
        at
org.jvnet.hk2.osgiadapter.OSGiModulesRegistryImpl.parseInhabitants(OSGiModulesRegistryImpl.java:191)
        at
com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.createHabitat(AbstractModulesRegistryImpl.java:140)
        at com.sun.enterprise.module.bootstrap.Main.createHabitat(Main.java:394)
        at org.jvnet.hk2.osgiadapter.HK2Main.start(HK2Main.java:118)
        at
org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:589)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:1458)
        at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:774)
        at org.jvnet.hk2.osgimain.Main.start(Main.java:104)
        at
org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:589)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:1458)
        at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:984)
        at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:263)
        at java.lang.Thread.run(Thread.java:637)
Caused by: org.osgi.framework.BundleException: Unresolved constraint in
bundle 37: package; (package=org.glassfish.api.branding)
        at org.apache.felix.framework.Felix.resolveBundle(Felix.java:3090)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:1439)
        at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:774)
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:137)
        ... 20 more




Jane Young wrote:
> Byron,
>
> Hudson continuous job is also having issue with server startup when
> running QL. See:
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/919/console
> Hudson node is using JDK version 1.6.0_10-b33.
>
> I take the same bundle, install it on my system (Mac and Solaris-Sparc)
> and passes QL tests.
>
> Can you take a look and let me know if this is related to the deadlock
> issue?
> The bundle is here:
> http://hudson.glassfish.org/job/gf-trunk-build-continuous/lastSuccessfulBuild/artifact/bundles/glassfish.zip
>
> Thanks,
> Jane
>
>
> Byron Nevins wrote:
>>
>> * We currently have deadlock issues coming from Logging code. You
>> should upgrade your JDK right away to > 1.6U10 The mightier your
>> computer is the more likely you will see the deadlock. We are
>> working on this problem RIGHT NOW.
>> * 90 seconds is an arbitrary time. We had to come up with a
>> one-size-fits-all number. We don't want to give a bogus message
>> when a user just has a very slow computer. The timeout is only
>> used in ususual cases -- like yours. When I see that happen, I
>> just ^C out after a "reasonable" amount of time and run "asadmin
>> uptime". You will immediately find out if the server is running.
>> * There is cold startup time and warm startup time. Cold is
>> currently very slow -- on my machine it is about 4X longer than
>> warm startup.
>> * Use "jps" (it is nestled in alongside javac, javap, etc.)
>> instead of "ps"
>> * Run "asadmin start-domain --verbose" and you will see all the
>> log messages. Enter a ctrl-backslash and then go look at
>> .../domain1/logs/jvm.log for a thread dump
>> * Say you already have some process using a port that GF wants --
>> the startup will fail pretty quickly and report exactly that
>> problem to you.
>>
>>
>>
>> Jonathan Benoit wrote:
>> i'm using Glassfish V3 nightly build (glassfish-v3-b47-04_27_2009.zip)
>> and i see this message trying to start Glassfish:
>>
>> % ./asadmin start-domain domain1
>>
>> Domain (domain1) did not respond in 90 seconds. It means it is still
>> coming up or it has failed to come up. Check server.log for details.
>>
>> I see nothing in server.log to indicate it failed to come up, nor do i
>> see any confirmation that server successfully started. I checked ps
>> output and i do see process started:
>>
>> 24195 pts/6 S 0:21 /usr/jdk/jdk1.6.0_04/bin/java -cp
>> /glassfishv3/glassfish/modules/glassfish.jar
>> -XX:+UnlockDiagnosticVMOptions -XX:MaxPermSize=192m -XX:NewRatio=2
>> -XX:+LogVMOutput
>> -XX:LogFile=/glassfishv3/glassfish/domains/domain1/logs/jvm.log
>> -Xmx512m -client -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>> -Djavax.net.ssl.trustStore=/glassfishv3/glassfish/domains/domain1/config/cacerts.jks
>> -Djavax.net.ssl.keyStore=/glassfishv3/glassfish/domains/domain1/config/keystore.jks
>> -Djava.security.policy=/glassfishv3/glassfish/domains/domain1/config/server.policy
>> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
>> -Dcom.sun.aas.instanceRoot=/glassfishv3/glassfish/domains/domain1
>> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>> -DWSIT_HOME=/glassfishv3/glassfish
>> -Djava.security.auth.login.config=/glassfishv3/glassfish/domains/domain1/config/login.conf
>> -Djava.endorsed.dirs=/glassfishv3/glassfish/lib/endorsed
>> -Dcom.sun.aas.installRoot=/glassfishv3/glassfish
>> -Djava.ext.dirs=/usr/jdk/jdk1.6.0_04/lib/ext:/usr/jdk/jdk1.6.0_04/jre/lib/ext:/glassfishv3/glassfish/domains/domain1/lib/ext:/glassfishv3/javadb/lib
>> com.sun.enterprise.glassfish.bootstrap.ASMain -domainname domain1
>> -asadmin-args start-domain,,,domain1 -instancename server -verbose
>> false -debug false -asadmin-classpath ./../modules/admin-cli.jar
>> -asadmin-classname com.sun.enterprise.admin.cli.AsadminMain -domaindir
>> /glassfishv3/glassfish/domains/domain1
>>
>> why do i see this message, (90 seconds seems like a long time to wait
>> for startup) and should I see a message in server.log confirming that
>> server did indeed start?
>> --
>> Byron Nevins Work 408-276-4089, Home 650-359-1290, Cell 650-784-4123 - Sun Microsystems, Inc.
>