Yes. We care more about the raw throughput. Our main goal is to achieve
higher maximum capacity through this project. Most of our traffic is small
keepalive messages, which has a timeout of 30 seconds. However the desirable
response time for game specific messages is no more than 500 milliseconds.
I looked at the GC log in more detail, the full GC looks better in NIO, but
we have a much worse minor GC time for YoungGen with NIO engine when compare
to old classic gameserver:
rserv_log_070817_125402:7436.600: [Full GC [PSYoungGen: 5488K->0K(61632K)]
[PSOldGen: 41294K->35274K(44992K)] 46782K->35274K(106624K) [PSPermGen:
16366K->16366K(26752K)], 0.1906040 secs]
rserv_log_070817_125402:7442.548: [GC [PSYoungGen: 45568K->1504K(61504K)]
80842K->36778K(106496K), 0.0078060 secs]
rserv_log_070817_125402:7448.310: [GC [PSYoungGen: 47072K->2112K(63168K)]
82346K->38638K(108160K), 0.0838490 secs]
rserv_log_070817_125402:7449.229: [GC [PSYoungGen: 49856K->5912K(53696K)]
86382K->44402K(98688K), 0.6270710 secs]
rserv_log_070817_125402:7450.578: [GC [PSYoungGen: 53656K->5856K(64192K)]
92146K->50203K(109184K), 1.0315760 secs]
rserv_log_070817_125402:7451.609: [Full GC [PSYoungGen: 5856K->0K(64192K)]
[PSOldGen: 44347K->38032K(48896K)] 50203K->38032K(113088K) [PSPermGen:
16374K->16374K(26240K)], 0.2052210 secs]
There are quiet some minor GCs in NIO is taking more than one second. The
average minor GC time is 0.206815. While the average time of minor GC for
none-NIO server is 0.0133541.
Full_GC_baseline avg / max duration in seconds: 0.39 / 0.58
Full_GC_NIO avg / max duration in seconds: 0.28 / 0.41
thanks,
Fay
On 8/21/07, charlie hunt <charlie.hunt_at_sun.com> wrote:
>
> Hi Fay,
>
> Are you most interested in "raw throughput" with this application with
> little or no interest in "pause time / responsiveness" ?
>
> Or, is "pause time / responsiveness" more important than throughput ?
>
> Do you have a threshold at which you want all your GC pause times to be
> less than?
>
> Let's start with that ... then we'll work on tuning gc for your NIO
> server.
>
> charlie ...
>
> Fay Zheng wrote:
> > Now we have a GC problem. The NIO server is using a lot more GC time:
> >
> > 1) GC data for NIO server :
> >
> >
> > GC collector: Name = 'PS Scavenge', Collections = 737, Total time
> > spent = 2 minutes
> >
> > GC collector: Name = 'PS MarkSweep', Collections = 59, Total time
> > spent = 16.414 seconds
> >
> > Committed memory: 97, 088 kbytes
> >
> >
> >
> > 1) GC data for none - NIO server :
> >
> > GC collector: Name = 'PS Scavenge', Collections = 1075, Total time
> > spent = 14.349 seconds
> >
> > GC collector: Name = 'PS MarkSweep', Collections = 17, Total time
> > spent = 6.511 seconds
> >
> > Committed memory: 102,464 kbytes
> >
> > And we are using the default GC settings from JVM 1.5.12.
> >
> > Any ideas on how to configure the GC for NIO engine will be greatly
> > appreciated?
> >
> > thanks,
> > Fay
> >
> > On 8/20/07, *Fay Zheng* <fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>> wrote:
> >
> > It seems that the fragmented packet behavior goes away after I set
> > tcpNoDelay to false explicitly in the socket connection
> > settings for both client and server.
> >
> > thanks,
> > Fay
> >
> >
> > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>> wrote:
> >
> > Any chance you can try your configuration on a platform other
> > than Linux
> > such as OpenSolaris (x86) or Windows ?
> >
> > Linux has a history of being a little trouble-some when it
> > comes to
> > working well with Java NIO.
> >
> > This is the first I have heard of someone observing the sort
> > of thing
> > you are describing where you read small portions of data at a
> > time.
> >
> > Can you isolate your testing system completely independent of
> > all other
> > LAN traffic, (i.e. isolate the test system to include only the
> > machine
> > your are testing on a load machine) ?
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Disable TcpNoDelay does not seem to help. We still receive
> > tons of
> > > fragmented packets on single read.
> > >
> > > Has anyone had similar experience before?
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> > wrote:
> > >
> > > Hi Fay,
> > >
> > > You could try it. It may help.
> > >
> > > If it does indeed improve things for you, let us know.
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Thank you for your help. I also noticed that there are
> > some TCP
> > > > specific behaviors. Looks like tcp connection is
> > sending 1 byte
> > > > (partial data) on one read, then the rest of the
> > request content on
> > > > next read. So we have to assembling the request. This
> > happens quiet
> > > > often during our load test. Should I enable "
> > tcpNoDelay"? Will it
> > > > help to turn off the behavior?
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/14/07, *charlie hunt* < charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com
> >>
> > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>> wrote:
> > > >
> > > > There's a couple places in the grizzly source code
> > you would
> > > need to
> > > > explicitly modify from that I looked at yesterday.
> > > >
> > > > Here's what I identified:
> > > >
> > > > 1.) In Controller.java , doSelect() method. Remove
> > this
> > > logic towards
> > > > the end of the method:
> > > > iterator = readyKeys.iterator();
> > > > long currentTime =
> > System.currentTimeMillis ();
> > > > while (iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > key = iterator.next();
> > > > selectionKeyHandler.expire
> > (key,currentTime);
> > > > }
> > > > *Note: In the trunk, Alexy just moved this logic to
> the
> > > > DefaultSelectionKeyHandler.
> > > >
> > > > 2.) If you are using the TCPSelectorHandler, in
> > onReadOps(),
> > > you don't
> > > > need the currentTime to initialized, you can just
> > set it to
> > > 0, or
> > > > remove
> > > > the currentTime and replace currentTime with 0L in
> the
> > > > selectionKeyHandler.register(key, currentTime)
> > call. Same thing
> > > > applies
> > > > to onWriteOps(). In onAcceptInterest(), you can
> > replace the
> > > call to
> > > > System.currentTimeMillis () with 0L.
> > > >
> > > > 3.) In DefaultSelectionKeyHandler.expire(), just
> > make the entire
> > > > method
> > > > a no-op. Just tell it to return as soon as it is
> > entered.
> > > >
> > > > I think that is it. That is all I could find.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Thank you! We've already override the expire
> > method in the
> > > > > DefaultSelectionKeyHandler to do nothing for our
> > application:
> > > > >
> > > > > DefaultSelectionKeyHandler keyHandler =
> > > > >
> > > > > *new* DefaultSelectionKeyHandler() { *public*
> *void*
> > > > > expire(SelectionKey key) { *return*;
> > > > >
> > > > > }
> > > > >
> > > > > };
> > > > >
> > > > > However this approach looks quiet expensive in
> > the profiling
> > > > data. For
> > > > > now I have to customize the grizzly source code
> > to remove the
> > > > expiring
> > > > > of key part.
> > > > >
> > > > > thanks,
> > > > > Fay
> > > > >
> > > > >
> > > > > On 8/13/07, *charlie hunt* < charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com
> >>
> > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>>> wrote:
> > > > >
> > > > > Hi Fay,
> > > > >
> > > > > At the moment, there is not a way to turn off
> > expiring
> > > of a
> > > > key.
> > > > >
> > > > > But, it is functionality we should have and I
> > know of
> > > > another project
> > > > > which will want this functionality disabled
> > as well.
> > > > >
> > > > > I will file an enhancement issue to add this
> > > functionality.
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > Is there a way to turn off expire of key,
> > since it
> > > is not
> > > > > applicable
> > > > > > to persistent connections:
> > > > > >
> > > > > > iterator = readyKeys.iterator ();
> > > > > >
> > > > > > while (iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > > >
> > > > > > key = iterator.next();
> > > > > >
> > > > > > selectionKeyHandler.expire(key);
> > > > > >
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Fay
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 8/10/07, *Fay Zheng* <
> > fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com
> >>
> > > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>
> > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>> <mailto:fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>
> > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>>>>
> > > > > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>
> > > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com
> >>
> > > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>>>
> > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>>
> > > > <mailto: fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>
> > > <mailto:fzheng1998_at_gmail.com
> > <mailto:fzheng1998_at_gmail.com>>>>>> wrote:
> > > > > >
> > > > > > Here's the command to run the
> application:
> > > > > >
> > > > > > java -server -verbose:gc -XX:+PrintGC
> > > > > > Details -XX:+PrintGCTimeStamps -
> > > > > >
> > Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > > -Dcom.sun.management.jmxremote
> > > > > > -
> Dcom.sun.management.jmxremote.port=9991
> > > > > >
> > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > > > - Duser.region=US -Xmx700m
> > com.pg.serv.room.Main
> > > > > >
> > > > > > The GC is almost 10% for NIO, 5% for
> > none NIO
> > > server.
> > > > > >
> > > > > > thansk,
> > > > > > Fay
> > > > > >
> > > > > > On 8/8/07, *charlie hunt* <
> > charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com
> >>
> > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com
> >>>>
> > > > > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>
> > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com
> >>
> > > > <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>
> > > <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>
> > > > <mailto:charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> > <mailto:charlie.hunt_at_sun.com>>>>>>
> > > > > wrote:
> > > > > >
> > > > > > In addition to "RG's" suggestion, I
> > would:
> > > > > >
> > > > > > 1.) Set use_direct_buffer =
> > false (Grizzly
> > > will use
> > > > > > HeapByteBuffer instead)
> > > > > > 2.) You might also reduce the
> > number of
> > > instances
> > > > of the
> > > > > > server down to
> > > > > > 2 or even 1. NIO should give you
> > much better
> > > > > scalability and
> > > > > > should
> > > > > > have no problems scaling to 12,000
> > connections.
> > > > > > 3.) If you can, I would also go to
> the
> > > latest JDK 6_02
> > > > > version.
> > > > > >
> > > > > > We might also need to do some
> > further fine
> > > tuning
> > > > of the
> > > > > JVM too.
> > > > > >
> > > > > > Could you share with us the full
> java
> > > command line
> > > > args you
> > > > > > are using?
> > > > > > It might also be useful to see the
> > output from
> > > > > > -XX:+PrintGCDetails when
> > > > > > you are running with NIO.
> > > > > >
> > > > > > hths,
> > > > > >
> > > > > > charlie ...
> > > > > >
> > > > > > Fay Zheng wrote:
> > > > > > > We add Grizzly Nio engine to one
> > of existing
> > > > server,
> > > > > which
> > > > > > used to be
> > > > > > > one thread per connection classic
> > model.
> > > During load
> > > > > test we
> > > > > > noticed
> > > > > > > that NIO server consumes twice
> > CPU power than
> > > > the classic
> > > > > > model when
> > > > > > > serving the same amount of
> > connections.
> > > > > > >
> > > > > > > Here's our configurations:
> > > > > > >
> > > > > > > Here are some statistics:
> > > > > > >
> > > > > > > Hardware: Dells 2950 with 16 GB
> > RAM, total 8
> > > > > processors, Intel(R)
> > > > > > > Xeon(R) CPU E5345 @ 2.33GHz ,
> Linux
> > > kernel 2.6
> > > > > > >
> > > > > > > Software: JDK v1.5.6
> > > > > > >
> > > > > > > There are 4 instance of the
> > server application
> > > > > configured on
> > > > > > one box
> > > > > > >
> > > > > > > Grizzly Configs:
> > > > > > >
> > > > > > > 100 max worker threads (8K buffer
> > size)
> > > > > > >
> > > > > > > 150 max output buffer pool size
> > (4K buffer
> > > size)
> > > > > > >
> > > > > > > use_direct_buffer = true
> > > > > > >
> > > > > > > selector timeout = 500
> > > > > > >
> > > > > > > Keep alive is true (persistent
> > connections)
> > > > > > >
> > > > > > >
> > > > > > > Tested with : 7200 client
> connections
> > > > > > >
> > > > > > > Total CPU usage for all 4
> > instances are
> > > 44.59% (with
> > > > > NIO engine)
> > > > > > >
> > > > > > > CPU usage is 24.76% (without Nio
> > engine)
> > > > > > >
> > > > > > > So far as memory, NIO server uses
> > slightly
> > > less
> > > > memory in
> > > > > > this case.
> > > > > > >
> > > > > > >
> > > > > > > The memory consumption could be
> > linear
> > > because
> > > > we keep
> > > > > lots
> > > > > > of state
> > > > > > > information for each connection
> > (user).
> > > Other than
> > > > > that, what
> > > > > > other
> > > > > > > parameters should I pay attention
> > to in
> > > order to
> > > > scale the
> > > > > > server to
> > > > > > > serve 10,000 or more connections
> > without
> > > burning
> > > > up CPU?
> > > > > > >
> > > > > > > The old server can serve 12000
> > connections
> > > with 4
> > > > > instance at
> > > > > > <= 50%
> > > > > > > CPU usage.
> > > > > > >
> > > > > > > Would it be better to run 2
> > instances of
> > > NIO server
> > > > > instead
> > > > > > of 4?
> > > > > > >
> > > > > > > Your opinion and suggestion is
> > greatly
> > > appreciated.
> > > > > > >
> > > > > > >
> > > > > > > thanks,
> > > > > > > Fay
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Charlie Hunt
> > > > > > Java Performance Engineer
> > > > > > 630.285.7708 x47708 (Internal)
> > > > > >
> > > > > > <
> http://java.sun.com/docs/performance/
> > > > > < http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > > >
> > <http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> >
> > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > > <mailto:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > > For additional commands, e-mail:
> > > > > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>> <mailto:
> > > > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > > <mailto:
> > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > < http://java.sun.com/docs/performance/
> > > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> >
> > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>
> > > >
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net> <mailto:
> > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > < http://java.sun.com/docs/performance/>
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > For additional commands, e-mail:
> > dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>>
> > >
> > >
> >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > <http://java.sun.com/docs/performance/>
> >
> >
> ---------------------------------------------------------------------
> >
> > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> > <mailto:dev-help_at_grizzly.dev.java.net>
> >
> >
> >
>
> --
>
> Charlie Hunt
> Java Performance Engineer
> 630.285.7708 x47708 (Internal)
>
> <http://java.sun.com/docs/performance/>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>
>