I'm trying to determine the cause of elevated CPU and Interrupt / Context-switch activity on our application server that occurs at non-peak traffic times and at times causes the application server to become non-responsive. Munin graphs of the CPU usage and Interrupts / Context-Switches are attached.
These spikes are odd because they occur during non-peak times and do not resolve themselves. So far, the only solution I have found is to restart glassfish, which usually makes the problem go away for 8-24 hours. Sometimes, these spikes do not affect load times, but sometimes the spike is more severe and the application server becomes non-responsive.
I have confirmed during these times that there are no other processes running that are using a significant amount of CPU; i.e. the CPU spikes are glassfish CPU spikes. IO activity is no higher than usual during there periods. Munin memory graphs showed no differences between problematic and non-problematic times.
The app is J2EE using JPA + Hibernate, no EJBs.
I'm looking for ideas / advice on tracking down the root cause. Thanks!
System:
- x-large high-cpu Amazon EC2 instance (7.5GB of memory, 8 cpus, 20 amazon "compute units")
- ubuntu 8.04, 64-bit
- dedicated to glassfish
Notable JVM options:
<jvm-options>-server</jvm-options>
<jvm-options>-Xmx2500m</jvm-options>
<jvm-options>-Xms2500m</jvm-options>
<jvm-options>-Xmn1000m</jvm-options>
<jvm-options>-Xss128k</jvm-options>
<jvm-options>-XX:+AggressiveOpts</jvm-options>
<jvm-options>-XX:+AggressiveHeap</jvm-options>
<jvm-options>-XX:+DisableExplicitGC</jvm-options>
<jvm-options>-XX:+UseParallelGC</jvm-options>
<jvm-options>-XX:+UseParallelOldGC</jvm-options>
<jvm-options>-XX:ParallelGCThreads=10</jvm-options>
<jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=false</jvm-options>
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
Notable Glassfish Settings:
<http-listener acceptor-threads="5" address="0.0.0.0" blocking-enabled="false" default-virtual-server="server" enabled="true" family="inet" id="http-listener-1" port="80" security-enabled="false" server-name="" xpowered-by="false"><property name="compression" value="on"/></http-listener>
<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="50" request-timeout-in-seconds="60" thread-count="250" thread-increment="10"/>
<keep-alive max-connections="256" thread-count="5" timeout-in-seconds="30"/>
<connection-pool max-pending-count="4096" queue-size-in-bytes="4096" receive-buffer-size-in-bytes="4096" send-buffer-size-in-bytes="8192"/>
<http-file-cache file-caching-enabled="true" file-transmission-enabled="false" globally-enabled="true" hash-init-size="0" max-age-in-seconds="60" max-files-count="1024" medium-file-size-limit-in-bytes="537600" medium-file-space-in-bytes="10485760" small-file-size-limit-in-bytes="2048" small-file-space-in-bytes="1048576"/>
[Message sent by forum member 'rwillie6' (rwillie6)]
http://forums.java.net/jive/thread.jspa?messageID=338592