users@glassfish.java.net

Re: Debugging Glassfish CPU usage

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Tue, 24 Mar 2009 11:07:25 +0100

Salut,

glassfish_at_javadesktop.org wrote:
> I'm trying to determine the cause of elevated CPU and Interrupt / Context-switch activity on our application server that occurs at non-peak traffic times and at times causes the application server to become non-responsive. Munin graphs of the CPU usage and Interrupts / Context-Switches are attached.
>
> These spikes are odd because they occur during non-peak times and do not resolve themselves. So far, the only solution I have found is to restart glassfish, which usually makes the problem go away for 8-24 hours. Sometimes, these spikes do not affect load times, but sometimes the spike is more severe and the application server becomes non-responsive.
>
> I have confirmed during these times that there are no other processes running that are using a significant amount of CPU; i.e. the CPU spikes are glassfish CPU spikes. IO activity is no higher than usual during there periods. Munin memory graphs showed no differences between problematic and non-problematic times.
>
> The app is J2EE using JPA + Hibernate, no EJBs.

any web related call? It looks like a possible JDK issue IMO (which JDK
are you using?). When it peak, can you grab a thread dump? I think the
NIO Selector might get into a spinning state...we are currently testing
a fix for JDK 7 b51 and if that work it will be backported to 6 urX.
Before I share a patch, I just want to make sure this is the issue.

Thanks

-- Jeanfrancois


>
> I'm looking for ideas / advice on tracking down the root cause. Thanks!
>
> System:
> - x-large high-cpu Amazon EC2 instance (7.5GB of memory, 8 cpus, 20 amazon "compute units")
> - ubuntu 8.04, 64-bit
> - dedicated to glassfish
>
> Notable JVM options:
> <jvm-options>-server</jvm-options>
> <jvm-options>-Xmx2500m</jvm-options>
> <jvm-options>-Xms2500m</jvm-options>
> <jvm-options>-Xmn1000m</jvm-options>
> <jvm-options>-Xss128k</jvm-options>
> <jvm-options>-XX:+AggressiveOpts</jvm-options>
> <jvm-options>-XX:+AggressiveHeap</jvm-options>
> <jvm-options>-XX:+DisableExplicitGC</jvm-options>
> <jvm-options>-XX:+UseParallelGC</jvm-options>
> <jvm-options>-XX:+UseParallelOldGC</jvm-options>
> <jvm-options>-XX:ParallelGCThreads=10</jvm-options>
> <jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=false</jvm-options>
> <jvm-options>-XX:MaxPermSize=192m</jvm-options>
> <jvm-options>-XX:NewRatio=2</jvm-options>
>
> Notable Glassfish Settings:
> <http-listener acceptor-threads="5" address="0.0.0.0" blocking-enabled="false" default-virtual-server="server" enabled="true" family="inet" id="http-listener-1" port="80" security-enabled="false" server-name="" xpowered-by="false"><property name="compression" value="on"/></http-listener>
> <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="50" request-timeout-in-seconds="60" thread-count="250" thread-increment="10"/>
> <keep-alive max-connections="256" thread-count="5" timeout-in-seconds="30"/>
> <connection-pool max-pending-count="4096" queue-size-in-bytes="4096" receive-buffer-size-in-bytes="4096" send-buffer-size-in-bytes="8192"/>
> <http-file-cache file-caching-enabled="true" file-transmission-enabled="false" globally-enabled="true" hash-init-size="0" max-age-in-seconds="60" max-files-count="1024" medium-file-size-limit-in-bytes="537600" medium-file-space-in-bytes="10485760" small-file-size-limit-in-bytes="2048" small-file-space-in-bytes="1048576"/>
> [Message sent by forum member 'rwillie6' (rwillie6)]
>
> http://forums.java.net/jive/thread.jspa?messageID=338592
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>