users@glassfish.java.net

My server is not spawning enough threads

From: <glassfish_at_javadesktop.org>
Date: Wed, 07 Oct 2009 10:01:18 PDT

Hello,

I am new to Glassfish. I deployed my webapp to GF V2 and it seems to work very well. The throughput is ~2X compared to Tomcat 6.0.18 :-)

The problem I am seeing is that although I am pumping 200 client request threads through my japex based test harness the server seems to only be running 2 request processing threads as reported by:

./asadmin get -m --iterations 10 --interval 5 server.http-service.server.http-listener-1.currentthreadcount-count server.http-service.server.http-listener-1.currentthreadsbusy-count

server.http-service.server.http-listener-1.currentthreadcount-count = 2
server.http-service.server.http-listener-1.currentthreadsbusy-count = 1

Note that I have set request-processing.thread-count=16 using:

./asadmin set server.http-service.request-processing.thread-count=16

Could it because all my requests are coming from multiple threads in the same client program (using http://japex.dev.java.net).

Any advice on how to determine the cause of the lack of parrallel threads?
TIA for your help.

Here is a dump of threads that I thought where relevant (had port 8080 in their name):

"httpSSLWorkerThread-8080-1" daemon prio=10 tid=0x00007f71b0142000 nid=0x41a8 in Object.wait() [0x0000000044993000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f71e05b5928> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
        at java.lang.Object.wait(Object.java:485)
        at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
        - locked <0x00007f71e05b5928> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)

   Locked ownable synchronizers:
        - None

"httpSSLWorkerThread-8080-0" daemon prio=10 tid=0x00007f71b08e5800 nid=0x41a7 runnable [0x0000000044890000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1166)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
        - locked <0x00007f71e2da2390> (a org.postgresql.core.v3.QueryExecutorImpl)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
        ....
        at org.springframework.ws.server.endpoint.adapter.PayloadEndpointAdapter.invoke(PayloadEndpointAdapter.java:48)
        at org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:221)
        at org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:168)
        at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:88)
        at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
        at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

   Locked ownable synchronizers:
        - None

"GrizzlyPipelineStatWorkerThread-8080-2" daemon prio=10 tid=0x00007f71bc26a000 nid=0x41a6 waiting on condition [0x0000000044791000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00007f71e065b180> (a .locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:107)

   Locked ownable synchronizers:
        - None

"GrizzlyPipelineStatWorkerThread-8080-1" daemon prio=10 tid=0x00007f71bd476000 nid=0x41a5 waiting on condition [0x0000000044690000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00007f71e065b180> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:107)

   Locked ownable synchronizers:
        - None

"GrizzlyPipelineStatWorkerThread-8080-0" daemon prio=10 tid=0x00007f71bd881000 nid=0x41a4 waiting on condition [0x000000004458f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00007f71e065b180> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:107)

   Locked ownable synchronizers:
        - None

"SelectorThread-8080" prio=10 tid=0x00007f71bcd6b800 nid=0x41a3 runnable [0x000000004428c000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00007f71e05b4e08> (a sun.nio.ch.Util$1)
        - locked <0x00007f71e05b4df0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00007f71e05b4d38> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295)
        - locked <0x00007f71e05b4e48> (a [Ljava.lang.Object;)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258)
        at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234)

   Locked ownable synchronizers:
        - None
[Message sent by forum member 'najmi' (farrukh_at_wellfleetsoftware.com)]

http://forums.java.net/jive/thread.jspa?messageID=367151