users@grizzly.java.net

Re: Comet - growing CPU usage

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Tue, 09 Jun 2009 11:45:29 -0400

Salut,


FredrikJ wrote:
> I have tried using the 1.0.29 version of Grizzly now and the CPU is looking
> better, but not optimal =/
>
> This is the CPU for our application using synchronous execution and Comet
> disabled (200 requests/second):
> http://www.nabble.com/file/p23939428/cpu_app_sync.png
>
> This is the CPU for our application using asynchronous execution and Comet
> enabled (200 requests/second):
> http://www.nabble.com/file/p23939428/cpu_app_comet.png
>
> The CPU looks better than before, but we are still using significantly (more
> than 2 times) more CPU and it grows steadily over time.


Can you elaborate why the CPU is an issue for you? Comet use more
threads when executing a normal request. You don't need more thread but
of course there will be more activity, hence CPU.

BTW looks like someone in tweeting on the same subject :-)

http://twitter.com/stnor/statuses/2088504828

>
> Granted, the CPU growing over time for this instance can be misuse of the
> API. I do get some of
> these:http://www.nabble.com/Nullpointer-in-Comet-td23919808.html ,
> exceptions in the logfile and it is unclear if that CometHandler is resumed
> properly... The exceptions occurs roughly 1 out of 300 requests.
>
> However, we did encounter a deadlock using the 1.0.29. We have not seen this
> before, but since the CPU usually went through the roof we were never able
> to run more than approx 20 minutes before crashing.
>
> I am including the stacktraces of the threads involved in the deadlock
> below. All other threads are blocked and waiting on one of these threads to
> release their locks (which of course will not happen). The threads and locks
> involved seems to be in the IO-part of Grizzly so I do not suspect that this
> is caused by our application. This seems critical to me since it is
> non-recoverable.
>

This issue has been fixed in 1.9.x but nobody ever reported it using
1.0.x. I suspect using GF v3 is no op for you, right?

Thanks

-- Jeanfrancois


>
> "SelectorThread-9080" - Thread t_at_62
> java.lang.Thread.State: BLOCKED on java.lang.Object_at_20583549 owned by:
> httpWorkerThread-9080-34
> at
> java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:111)
> at
> java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:168)
> at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:156)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:131)
> - locked java.util.HashSet_at_63581e9d
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked sun.nio.ch.Util$1_at_7efcd3b4
> - locked java.util.Collections$UnmodifiableSet_at_8af0f8e
> - locked sun.nio.ch.EPollSelectorImpl_at_120970c3
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295)
> - locked [Ljava.lang.Object;@561da30
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234)
>
>
> "httpWorkerThread-9080-34" - Thread t_at_148
> java.lang.Thread.State: BLOCKED on java.util.HashSet_at_2e42085f owned by:
> CometSelector
> at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:71)
> at
> java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:56)
> - locked sun.nio.ch.SelectionKeyImpl_at_1838b916
> at
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:207)
> - locked java.lang.Object_at_20583549
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
> - locked java.lang.Object_at_23cf84c9
> at sun.nio.ch.SocketAdaptor.close(SocketAdaptor.java:352)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.cancelKey(SelectorThread.java:1573)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.finishConnection(DefaultReadTask.java:430)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.terminate(DefaultReadTask.java:375)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.terminate(AsyncReadTask.java:177)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.manageKeepAlive(AsyncReadTask.java:93)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:279)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.doTask(AsyncReadTask.java:72)
> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
> at
> com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:116)
>
>
> "CometSelector" - Thread t_at_118
> java.lang.Thread.State: BLOCKED on java.lang.Object_at_7133acb2 owned by:
> httpWorkerThread-9080-65
> at
> java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:111)
> at
> java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:168)
> at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:156)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:131)
> - locked java.util.HashSet_at_2e42085f
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:62)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked sun.nio.ch.Util$1_at_7020c12
> - locked java.util.Collections$UnmodifiableSet_at_47a16d1a
> - locked sun.nio.ch.EPollSelectorImpl_at_5ed7a9fb
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> com.sun.enterprise.web.connector.grizzly.comet.CometSelector$1.run(CometSelector.java:131)
>
>
> "httpWorkerThread-9080-65" - Thread t_at_234
> java.lang.Thread.State: BLOCKED on java.util.HashSet_at_63581e9d owned by:
> SelectorThread-9080
> at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:71)
> at
> java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:56)
> - locked sun.nio.ch.SelectionKeyImpl_at_5e12f46b
> at
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:207)
> - locked java.lang.Object_at_7133acb2
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
> - locked java.lang.Object_at_38e56634
> at sun.nio.ch.SocketAdaptor.close(SocketAdaptor.java:352)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.cancelKey(SelectorThread.java:1573)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.finishConnection(DefaultReadTask.java:430)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.terminate(DefaultReadTask.java:375)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.terminate(AsyncReadTask.java:177)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.manageKeepAlive(AsyncReadTask.java:93)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:279)
> at
> com.sun.enterprise.web.connector.grizzly.AsyncReadTask.doTask(AsyncReadTask.java:72)
> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
> at
> com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:116)
>
>
>
>