I'm testing throughput/performance of an app that is attempting to
utilize the highly multiplexed nature of Grizzly. JAX-RS Web Services
are being served up with Jersey.
The HTTP configuration in glassfish is configured with:
Request Processing
=============
Thread Count: 100
Initial Thread Count: 20
Thread Increment: 5
Request Timeout: 30
Buffer Length: 8192
Keep Alive
========
Thread Count: 1
Max Connections: 250
Time Out: 30
Connection Pool
============
Max Pending Count: 4096
Queue Size: 4096
Receive Buffer Size: 4096
Send Buffer Size: 8192
HTTP Listener (non-SSL)
=================
Acceptor Threads: 10
Blocking: Disabled (unchecked)
Comet is enabled
The glassfish domain is being run on my dev box:
Glassfish v2.1 FCS
Win XP SP3, JDK 1.6.0_12 (32bit)
Max heap size 1024
Max perm gen is 256
Now, when testing I've got three to four other servers (solaris, linux
boxes) running many simultaneous threads that are all making HTTP
requests. When I've got about 40 simultaneous threads going, everything
is fine, but once I go to about 45-50, the http worker threads seem to
"freeze" up.
There are no errors in my app or the domains log whatsoever (no log
entries at all when it freezes up)
I'm monitoring the domain with jvisualvm that ships with JDK 1.6, the
heap never exceeds 160MB (which is great), perm gen never exceed 80MB
and all the http worker threads (yes, all of the ones that were active)
turn green, which visualvm labels as "running" (note that when they are
not doing any work, they are yellow in the "wait" state
I'm not a grizzly expert, so im not sure what configuration parameters
should be tweaked to help scale for more concurrent connections.
My requests are XML PUT requests, they are about 27KB in size each. The
processing time on the server is very short lived (at most 100ms per
request, but 99% of the time should be < 20ms).
Any idea on what the bottleneck could be in grizzly config?