I am stress testing an application that should be ready to handle a high number of small requests. At this moment, the bottleneck seems to be in closing threads in the connection pool. I am trying to find the optimum number of request processing threads, by monitoring the currentthreadcount and the currentthreadbusy values:
asadmin get -m --iterations 100 --interval 5 server.http-service.server.http-listener-1.currentthreadcount-count server.http-service.server.http-listener-1.currentthreadsbusy-count
I am currently using a pool with 64 threads on a quad core server, which seems too high to me. But after some time, the currentthreadbusy value reaches 64 and it stays there. The CPU load is low, at that moment.
When generating a thread dump, it turns out that there are 64 threads like this:
Thread "httpSSLWorkerThread-80-61" thread-id 170 thread-stateRUNNABLERunning in native
at: sun.nio.ch.FileDispatcher.preClose0(Native Method)
at: sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41)
at: sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:684)
at: java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:201)
at: java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at: sun.nio.ch.SocketAdaptor.close(SocketAdaptor.java:352)
at: com.sun.enterprise.web.connector.grizzly.SelectorThread.cancelKey(SelectorThread.java:1573)
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.finishConnection(DefaultReadTask.java:430)
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.terminate(DefaultReadTask.java:375)
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.manageKeepAlive(DefaultReadTask.java:310)
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:279)
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at: com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Those threads don't stay in preClose0() forever, but long enough (without consuming cpu resources) to slow down the processing.
In another message on this forum, I read something similar (
http://forums.java.net/jive/thread.jspa?messageID=339116񒲬). The problem seemed to be memory-related there, but in our case the gc is doing nothing unusual. However, I tried the same trick as in this post: setting the number of acceptor threads to 1 instead of 4 dramatically increases the performance. When taking threaddumps, I hardly see any threads in preClose0 anymore. It seems that connection are reused immediately.
Increasing the number of acceptor threads reintroduces the problem. Worst-case we have to use the application with one single acceptor thread, but I would feel more confident if I know the cause of this behavior - any help is appreciated.
Some more info:
uname -a
Linux xyz 2.6.27.19-170.2.35.fc10.x86_64 #1 SMP Mon Feb 23 13:00:23 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
GlassFish 2.1
Thanks,
- Johan
[Message sent by forum member 'johanvos' (johanvos)]
http://forums.java.net/jive/thread.jspa?messageID=344696