Hi,
We have Sun Java System Application Server Platform Edition 9.0_01 (build ) on Production Environment which is running on operating system
Red Hat Enterprise Linux ES release 4 (Nahant Update 6). This Application Server
Linux instance is running on VMWare with host configuration 16GB RAM,
Dual Core AMD Opteron 2220 2.8GHz. The Application Server Linux Instance is allocated with 3GB RAM, and single core processor.
We are using MySQL 5.0.45 community edition for the database. It is configured on another server with 1GB RAM and single core processor.
We have JVM options as follows for the app server,
Max Heap Memory(Xmx) : 1536
Min Heap Memory(Xms): 512M
MaxPermGeSpace:128M
-server
[b]
ISSUE:[/b] Application server is going down for every 36 hours , we have monitoring tool installed on the server to alert when heap memory reaches maximum.
Application has a maximum of 25 concurrent users. The server is going down even on the weekends when none of the users are accessing the application.
However, there are no "Out of memory" or "Perm Gen" errors trapped either in our application or server log files.
We see only the following entries in "server.log".
------------------------------------------------------------------------------------------------------------------------
[#|2008-04-25T11:43:28.428+0100|INFO|sun-appserver-pe9.0|javax.enterprise.tools.launcher|_ThreadID=10;_ThreadName=main;|
/usr/java/latest/bin/java
-client
-Dcom.sun.aas.defaultLogFile=/opt/SUNWappserver/domains/domain1/logs/server.log
-Dcom.sun.aas.verboseMode=true
-Djava.endorsed.dirs=/opt/SUNWappserver/lib/endorsed
-Djava.security.policy=/opt/SUNWappserver/domains/domain1/config/server.policy
-Djava.security.auth.login.config=/opt/SUNWappserver/domains/domain1/config/login.conf
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dsun.rmi.dgc.client.gcInterval=3600000
-Djavax.net.ssl.keyStore=/opt/SUNWappserver/domains/domain1/config/warranty.jks
-Djavax.net.ssl.trustStore=/opt/SUNWappserver/domains/domain1/config/cacerts.jks
-Djavax.net.ssl.keyStorePassword=csLffLmS01628!
-Djava.ext.dirs=/usr/java/latest/jre/lib/ext:/opt/SUNWappserver/domains/domain1/lib/ext:/opt/SUNWappserver/javadb/lib
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
-Dcom.sun.enterprise.taglisteners=jsf-impl.jar
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false
-Dcom.sun.aas.classloader.optionalOverrideableChain=appserv-ws.jar,commons-logging.jar,commons-launcher.jar
-Dcom.sun.aas.classloader.appserverChainJars=admin-cli.jar,admin-cli-ee.jar,dbschema.jar,j2ee-svc.jar
-Dcom.sun.aas.classloader.serverClassPath.ee=%HADB_HOME%/lib/hadbjdbc4.jar,/opt/SUNWappserver/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,%HADB_HOME%/lib/dbstate.jar,%HADB_HOME%/lib/hadbm.jar,%HADB_HOME%/lib/hadbmgt.jar,/opt/SUNWmfwk/lib/mfwk_instrum_tk.jar
-Dcom.sun.aas.configName=server-config
-Ddomain.name=
-Djmx.invoke.getters=true
-Dcom.sun.aas.promptForIdentity=true
-Dcom.sun.aas.classloader.optionalOverrideableChain.ee=
-Dcom.sun.aas.instanceRoot=/opt/SUNWappserver/domains/domain1
-Dcom.sun.aas.domainName=
-Dcom.sun.aas.classloader.sharedChainJars=javaee.jar,/usr/java/latest/lib/tools.jar,install/applications/jmsra/imqjmsra.jar,commons-launcher.jar,/opt/SUNWappserver/imq/lib/jaxm-api.jar,/opt/SUNWappserver/imq/lib/fscontext.jar,/opt/SUNWappserver/imq/lib/imqbroker.jar,/opt/SUNWappserver/imq/lib/imqjmx.jar,/opt/SUNWappserver/imq/lib/imqxm.jar,/opt/SUNWappserver/lib/ant/lib/ant.jar,appserv-ws.jar,mail.jar,jsf-api.jar,jsf-impl.jar,appserv-jstl.jar,appserv-env.jar,jmxremote_optional.jar,/opt/SUNWappserver/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,commons-logging.jar,activation.jar,appserv-rt.jar,appserv-admin.jar,appserv-cmp.jar
-Dcom.sun.aas.classloader.sharedChainJars.ee=appserv-se.jar,appserv-ee.jar,%HADB_HOME%/lib/dbstate.jar,%HADB_HOME%/lib/hadbjdbc4.jar,jgroups-all.jar,/opt/SUNWmfwk/lib/mfwk_instrum_tk.jar
-Djava.util.logging.manager=com.sun.enterprise.server.logging.ServerLogManager
-Dcom.sun.enterprise.overrideablejavaxpackages=javax.xml.bind,javax.help,javax.portlet
-Dcom.sun.aas.classloader.serverClassPath=/opt/SUNWappserver/lib/install/applications/jmsra/imqjmsra.jar,/opt/SUNWappserver/imq/lib/jaxm-api.jar,/opt/SUNWappserver/imq/lib/fscontext.jar,/opt/SUNWappserver/imq/lib/imqbroker.jar,/opt/SUNWappserver/imq/lib/imqjmx.jar,/opt/SUNWappserver/lib/ant/lib/ant.jar,/opt/SUNWappserver/lib/SUNWjdmk/5.1/lib/jdmkrt.jar
-Dcom.sun.aas.classloader.appserverChainJars.ee=
-Dcom.sun.aas.configRoot=/opt/SUNWappserver/config
-Djava.library.path=/usr/java/jdk1.6.0_03/jre/lib/amd64/server:/usr/java/jdk1.6.0_03/jre/lib/amd64:/usr/java/jdk1.6.0_03/jre/../lib/amd64:/opt/SUNWappserver/lib:/usr/lib/lwp:/usr/java/jdk1.6.0_03/jre/lib/amd64/server:/usr/java/jdk1.6.0_03/jre/lib/amd64:/usr/java/jdk1.6.0_03/jre/../lib/amd64:/opt/SUNWappserver/lib:/opt/SUNWappserver/lib:/opt/SUNWappserver/lib::/usr/java/packages/lib/amd64:/lib:/usr/lib
-Dcom.sun.aas.instanceName=server
-Dcom.sun.aas.processLauncher=SE
-Dcom.sun.aas.installRoot=/opt/SUNWappserver
-Dcom.sun.aas.ClassPathPrefix=
-Dcom.sun.aas.ClassPathSuffix=
-Dcom.sun.aas.ServerClassPath=
-cp
:/opt/SUNWappserver/lib/appserv-launch.jar
com.sun.enterprise.server.PELaunch
stop
verbose|#]
[#|2008-04-25T11:43:29.572+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=10;_ThreadName=main;|sending notification to server...server|#]
[#|2008-04-25T11:43:29.572+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=10;_ThreadName=main;|sending notification to server...server|#]
[#|2008-04-25T11:43:29.635+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.web|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|WEB0303: Stopping Sun-Java-System/Application-Server.|#]
[#|2008-04-25T11:43:29.655+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.web|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;8080;|WEB0713: Stopping Sun-Java-System/Application-Server HTTP/1.1 on 8080|#]
[#|2008-04-25T11:43:29.695+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.web|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;8181;|WEB0713: Stopping Sun-Java-System/Application-Server HTTP/1.1 on 8181|#]
[#|2008-04-25T11:43:29.730+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.web|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;4848;|WEB0713: Stopping Sun-Java-System/Application-Server HTTP/1.1 on 4848|#]
[#|2008-04-25T11:43:32.640+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.web|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|WebModule[/warranty] ServletContext.log():QuartzInitializer: Quartz Scheduler successful shutdown.|#]
[#|2008-04-25T11:43:33.559+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|CORE5051: Shutting down all J2EE applications ...|#]
[#|2008-04-25T11:43:33.561+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.container.ejb|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;2008/04/25 11:43:33;|EJB5122:EJB Timer Service shutdown at [2008/04/25 11:43:33]|#]
[#|2008-04-25T11:43:33.561+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|CORE 5060 : Stopping all J2EE Connector 1.5 Compliant resource adapters ...|#]
[#|2008-04-25T11:43:33.565+0100|INFO|sun-appserver-pe9.0|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=29;_ThreadName=Thread-804;|MQJMSRA_RA1101: SJSMQ JMSRA stopping...|#]
[#|2008-04-25T11:43:33.649+0100|WARNING|sun-appserver-pe9.0|javax.enterprise.system.stream.err|_ThreadID=29;_ThreadName=Thread-804;_RequestID=d70025ed-7ceb-47cb-851e-609ac1250cf0;|
[25/Apr/2008:11:43:33 BST] [B1047]: Shutting down broker...
|#]
[#|2008-04-25T11:43:33.650+0100|WARNING|sun-appserver-pe9.0|javax.enterprise.system.stream.err|_ThreadID=29;_ThreadName=Thread-804;_RequestID=d70025ed-7ceb-47cb-851e-609ac1250cf0;|[25/Apr/2008:11:43:33 BST] [B1077]: Broadcast good-bye to all connections ...
|#]
[#|2008-04-25T11:43:33.650+0100|WARNING|sun-appserver-pe9.0|javax.enterprise.system.stream.err|_ThreadID=29;_ThreadName=Thread-804;_RequestID=d70025ed-7ceb-47cb-851e-609ac1250cf0;|[25/Apr/2008:11:43:33 BST] [B1078]: Flushing good-bye messages ...
|#]
[#|2008-04-25T11:43:33.961+0100|WARNING|sun-appserver-pe9.0|javax.enterprise.system.stream.err|_ThreadID=29;_ThreadName=Thread-804;_RequestID=d70025ed-7ceb-47cb-851e-609ac1250cf0;|[25/Apr/2008:11:43:33 BST] [B1063]: Done
|#]
[#|2008-04-25T11:43:40.785+0100|WARNING|sun-appserver-pe9.0|javax.enterprise.system.stream.err|_ThreadID=29;_ThreadName=Thread-804;_RequestID=d70025ed-7ceb-47cb-851e-609ac1250cf0;|[25/Apr/2008:11:43:40 BST] [B1048]: Shutdown of broker complete.
|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=29;_ThreadName=Thread-804;|MQJMSRA_RA1101: SJSMQ JMSRA stopped.|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.enterprise.resource.resourceadapter|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;jmsra;|RAR7094: jmsra shutdown successful.|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.enterprise.resource.resourceadapter|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;__ds;|RAR7094: __ds shutdown successful.|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.enterprise.resource.resourceadapter|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;__xa;|RAR7094: __xa shutdown successful.|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|CORE 5061 : Resource adapters Stop() Complete|#]
[#|2008-04-25T11:43:40.792+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|CORE5052: Application shutdown complete.|#]
[#|2008-04-25T11:43:40.836+0100|INFO|sun-appserver-pe9.0|org.apache.catalina.loader.WebappClassLoader|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|Illegal access: this web application instance has been stopped already (the eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact)|#]
[#|2008-04-25T11:43:40.836+0100|INFO|sun-appserver-pe9.0|org.apache.catalina.loader.WebappClassLoader|_ThreadID=28;_ThreadName=RMI TCP Connection(3073)-127.0.0.1;|Illegal access: this web application instance has been stopped already (the eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact)|#]
[#|2008-04-25T11:43:41.384+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=10;_ThreadName=main;|Server shutdown complete.|#]
[#|2008-04-25T11:43:41.384+0100|INFO|sun-appserver-pe9.0|javax.enterprise.system.core|_ThreadID=10;_ThreadName=main;|Server shutdown complete.|#]
------------------------------------------------------------------------------------------------------------------------
The same application on dev environment on same RHL version with Sun Java System Application Server 9.1_01 (build b09d-fcs) with 4GB RAM (both app server & MySQL DB on this machine) is running absolutely fine.
Any help or pointers in this regard are greatly appreciated.
Regards,
Viswa
[Message sent by forum member 'vish_mvss2008' (vish_mvss2008)]
http://forums.java.net/jive/thread.jspa?messageID=271949