I have a set of standalone instances that when a new version of our application deployment is made, the instance under the node agent seem to simply freeze.....sometimes. 8 out of 10 times, the deployment works fine, but when the 2 fail, the entire instance freezes and I cannot even get any more detail from the process.
[b]1.[/b] Deploy the war to the target
[b]2.[/b] The instance seems to sync with "Synchronization for someapp_instance is complete. Total time spent 3304 milli second(s)"
[b]3.[/b] I get this "[#|2009-09-08T17:42:47.564-0400|INFO|sun-appserver2.1|javax.enterprise.system.stream.out|_ThreadID=42;_ThreadName=RMI TCP Connection(99)-10.1.6.30;|
SharedSecrets.getJavaNetAccess()=java.net.URLClassLoader$7_at_38a68a10|#]
"
[b]4.[/b] One of the servlets that is supposed to start at startup (via web.xml) starts it's work....but it then freezes...
[b]5.[/b] I try to get more info from the process with jps, but jps thinks its not there:
# jps
5007 Jps
559 Bootstrap
555 NodeAgentMain
but ps says it's there, see? pid 4895 is the stuck process and 555 is the nodeagent:
# ps -elf | grep -i jdk
0 O root 4895 555 0 39 20 ? 7291432 16:20:35 ? 4:24 /opt//jdk1.6.0_14/bin/sparcv
0 S root 555 1 0 40 20 ? 140964 ? Aug 30 ? 15:03 /opt/jdk1.6.0_14/jre/../bin
[b]6.[/b] Jstack doesn't think it's there either:
# jstack 4895
4895: Unable to open door: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
# jstack -F 4895
Attaching to process ID 4895, please wait...
Error attaching to process: Attach failed : No such process
# ps -elf | grep -i jdk
0 S root 4895 555 0 39 20 ? 7291432 ? 16:20:35 ? 4:42 /opt/jdk1.6.0_14/bin/sparcv
0 S root 555 1 0 40 20 ? 140964 ? Aug 30 ? 15:04 /opt//jdk1.6.0_14/jre/../b
[b]7.[/b] After about 45 mins (ugh) the process goes zombie:
# /usr/ucb/ps -auxww | grep -i 4895
root 4895 0.0 0.0 0 0 Z 7:12 <defunct>
root 5089 0.0 0.0 1688 1336 pts/4 S 17:24:08 0:00 grep -i 4895
[b]8.[/b] And then simply dies.
[b]9.[/b] I re-deploy the exact same war to the instance and it comes up, but it provides this message first:
[#|2009-09-08T17:25:53.685-0400|SEVERE|sun-appserver2.1|javax.enterprise.system.core.classloading|_ThreadID=15;_ThreadName=RMI TCP Connection(14)-10.1.6.30;_RequestID=16db1f0e-ea8b-4e25-a4f3-51d20de85fd5;|LDR5008: ConfigException :
com.sun.enterprise.config.ConfigException: Failed to load deployment descriptor for: someapp_instance
cause: No Such Web Module
Caused by: com.sun.enterprise.config.ConfigException: No Such Web Module
at com.sun.enterprise.instance.WebModulesManager.getWebModule(WebModulesManager.java:108)
at com.sun.enterprise.instance.WebModulesManager.isSystemPredeployed(WebModulesManager.java:217)
[#|2009-09-08T17:25:53.697-0400|WARNING|sun-appserver2.1|javax.enterprise.system.core.classloading|_ThreadID=15;_ThreadName=RMI TCP Connection(14)-10.1.6.30;_RequestID=16db1f0e-ea8b-4e25-a4f3-51d20de85fd5;|Exception during handleApplicationEvent
java.lang.NullPointerException
at com.sun.enterprise.webservice.ApplicationLoaderEventListenerImpl.handleApplicationEvent(ApplicationLoaderEventListenerImpl.java:84)
at com.sun.enterprise.server.event.ApplicationLoaderEventNotifier.notifyListeners(ApplicationLoaderEventNotifier.java:154)
The only guesses I have right now are:
- The classloader during deployments is broken somehow?
- The application, when being shutdown, is not completely clean when the new application tries to start up?
- The DAS (web) deployment process is somehow not working right (corrupting the deployment?). I found this URL
http://forums.java.net/jive/thread.jspa?messageID=339427 which says to change INSTANCE-SYNC-JVM-OPTIONS.....but I don't understand how to change it. Do I change it on the DAS, in the node agent, in the instance?
Thanks.
Server v2.1 (9.1.1) (build b60e-fcs)
SunOS hj-t5120-h 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5120 Solaris
java version "1.6.0_14"
[Message sent by forum member 'h1d3m3' (h1d3m3_at_yahoo.com)]
http://forums.java.net/jive/thread.jspa?messageID=363292