dev@grizzly.java.net

Re: High CPU usage when using Grizzly NIO engine

From: charlie hunt <charlie.hunt_at_sun.com>
Date: Tue, 21 Aug 2007 10:25:01 -0500

This is good news!

Glad to hear you have found a resolution.

You will probably need to explicitly make a call to
TCPSelectorHandler.setTcpNoDelay(false) when you configure Grizzly.

I will add a comment to the code and JavaDoc for TCPSelectorHandler
which reflects your findings.

charlie ...


Fay Zheng wrote:
> It seems that the fragmented packet behavior goes away after I set
> tcpNoDelay to false explicitly in the socket connection settings for
> both client and server.
>
> thanks,
> Fay
>
>
> On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> wrote:
>
> Any chance you can try your configuration on a platform other than
> Linux
> such as OpenSolaris (x86) or Windows ?
>
> Linux has a history of being a little trouble-some when it comes to
> working well with Java NIO.
>
> This is the first I have heard of someone observing the sort of thing
> you are describing where you read small portions of data at a time.
>
> Can you isolate your testing system completely independent of all
> other
> LAN traffic, (i.e. isolate the test system to include only the machine
> your are testing on a load machine) ?
>
> charlie ...
>
> Fay Zheng wrote:
> > Disable TcpNoDelay does not seem to help. We still receive tons of
> > fragmented packets on single read.
> >
> > Has anyone had similar experience before?
> >
> > thanks,
> > Fay
> >
> >
> > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>> wrote:
> >
> > Hi Fay,
> >
> > You could try it. It may help.
> >
> > If it does indeed improve things for you, let us know.
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Thank you for your help. I also noticed that there are
> some TCP
> > > specific behaviors. Looks like tcp connection is sending 1
> byte
> > > (partial data) on one read, then the rest of the request
> content on
> > > next read. So we have to assembling the request. This
> happens quiet
> > > often during our load test. Should I enable " tcpNoDelay"?
> Will it
> > > help to turn off the behavior?
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/14/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>> wrote:
> > >
> > > There's a couple places in the grizzly source code you
> would
> > need to
> > > explicitly modify from that I looked at yesterday.
> > >
> > > Here's what I identified:
> > >
> > > 1.) In Controller.java , doSelect() method. Remove this
> > logic towards
> > > the end of the method:
> > > iterator = readyKeys.iterator();
> > > long currentTime = System.currentTimeMillis ();
> > > while (iterator.hasNext() &&
> > selectorHandler.isOpen()) {
> > > key = iterator.next();
> > > selectionKeyHandler.expire
> (key,currentTime);
> > > }
> > > *Note: In the trunk, Alexy just moved this logic to the
> > > DefaultSelectionKeyHandler.
> > >
> > > 2.) If you are using the TCPSelectorHandler, in
> onReadOps(),
> > you don't
> > > need the currentTime to initialized, you can just set
> it to
> > 0, or
> > > remove
> > > the currentTime and replace currentTime with 0L in the
> > > selectionKeyHandler.register(key, currentTime)
> call. Same thing
> > > applies
> > > to onWriteOps(). In onAcceptInterest(), you can
> replace the
> > call to
> > > System.currentTimeMillis () with 0L.
> > >
> > > 3.) In DefaultSelectionKeyHandler.expire(), just make
> the entire
> > > method
> > > a no-op. Just tell it to return as soon as it is entered.
> > >
> > > I think that is it. That is all I could find.
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Thank you! We've already override the expire method
> in the
> > > > DefaultSelectionKeyHandler to do nothing for our
> application:
> > > >
> > > > DefaultSelectionKeyHandler keyHandler =
> > > >
> > > > *new* DefaultSelectionKeyHandler() { *public* *void*
> > > > expire(SelectionKey key) { *return*;
> > > >
> > > > }
> > > >
> > > > };
> > > >
> > > > However this approach looks quiet expensive in the
> profiling
> > > data. For
> > > > now I have to customize the grizzly source code to
> remove the
> > > expiring
> > > > of key part.
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/13/07, *charlie hunt* < charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>> wrote:
> > > >
> > > > Hi Fay,
> > > >
> > > > At the moment, there is not a way to turn off
> expiring
> > of a
> > > key.
> > > >
> > > > But, it is functionality we should have and I
> know of
> > > another project
> > > > which will want this functionality disabled as
> well.
> > > >
> > > > I will file an enhancement issue to add this
> > functionality.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Is there a way to turn off expire of key, since it
> > is not
> > > > applicable
> > > > > to persistent connections:
> > > > >
> > > > > iterator = readyKeys.iterator();
> > > > >
> > > > > while (iterator.hasNext() &&
> > selectorHandler.isOpen()) {
> > > > >
> > > > > key = iterator.next();
> > > > >
> > > > > selectionKeyHandler.expire(key);
> > > > >
> > > > >
> > > > >
> > > > > thanks,
> > > > >
> > > > > Fay
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 8/10/07, *Fay Zheng* < fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>>> wrote:
> > > > >
> > > > > Here's the command to run the application:
> > > > >
> > > > > java -server -verbose:gc -XX:+PrintGC
> > > > > Details -XX:+PrintGCTimeStamps -
> > > > > Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > -Dcom.sun.management.jmxremote
> > > > > - Dcom.sun.management.jmxremote.port=9991
> > > > >
> -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > > - Duser.region=US -Xmx700m
> com.pg.serv.room.Main
> > > > >
> > > > > The GC is almost 10% for NIO, 5% for none NIO
> > server.
> > > > >
> > > > > thansk,
> > > > > Fay
> > > > >
> > > > > On 8/8/07, *charlie hunt* <
> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>>
> > > > wrote:
> > > > >
> > > > > In addition to "RG's" suggestion, I would:
> > > > >
> > > > > 1.) Set use_direct_buffer =
> false (Grizzly
> > will use
> > > > > HeapByteBuffer instead)
> > > > > 2.) You might also reduce the number of
> > instances
> > > of the
> > > > > server down to
> > > > > 2 or even 1. NIO should give you much
> better
> > > > scalability and
> > > > > should
> > > > > have no problems scaling to 12,000
> connections.
> > > > > 3.) If you can, I would also go to the
> > latest JDK 6_02
> > > > version.
> > > > >
> > > > > We might also need to do some further fine
> > tuning
> > > of the
> > > > JVM too.
> > > > >
> > > > > Could you share with us the full java
> > command line
> > > args you
> > > > > are using?
> > > > > It might also be useful to see the
> output from
> > > > > -XX:+PrintGCDetails when
> > > > > you are running with NIO.
> > > > >
> > > > > hths,
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > We add Grizzly Nio engine to one of
> existing
> > > server,
> > > > which
> > > > > used to be
> > > > > > one thread per connection classic
> model.
> > During load
> > > > test we
> > > > > noticed
> > > > > > that NIO server consumes twice CPU
> power than
> > > the classic
> > > > > model when
> > > > > > serving the same amount of connections.
> > > > > >
> > > > > > Here's our configurations:
> > > > > >
> > > > > > Here are some statistics:
> > > > > >
> > > > > > Hardware: Dells 2950 with 16 GB RAM,
> total 8
> > > > processors, Intel(R)
> > > > > > Xeon(R) CPU E5345 @ 2.33GHz , Linux
> > kernel 2.6
> > > > > >
> > > > > > Software: JDK v1.5.6
> > > > > >
> > > > > > There are 4 instance of the server
> application
> > > > configured on
> > > > > one box
> > > > > >
> > > > > > Grizzly Configs:
> > > > > >
> > > > > > 100 max worker threads (8K buffer size)
> > > > > >
> > > > > > 150 max output buffer pool size (4K
> buffer
> > size)
> > > > > >
> > > > > > use_direct_buffer = true
> > > > > >
> > > > > > selector timeout = 500
> > > > > >
> > > > > > Keep alive is true (persistent
> connections)
> > > > > >
> > > > > >
> > > > > > Tested with : 7200 client connections
> > > > > >
> > > > > > Total CPU usage for all 4 instances are
> > 44.59% (with
> > > > NIO engine)
> > > > > >
> > > > > > CPU usage is 24.76% (without Nio engine)
> > > > > >
> > > > > > So far as memory, NIO server uses
> slightly
> > less
> > > memory in
> > > > > this case.
> > > > > >
> > > > > >
> > > > > > The memory consumption could be linear
> > because
> > > we keep
> > > > lots
> > > > > of state
> > > > > > information for each connection (user).
> > Other than
> > > > that, what
> > > > > other
> > > > > > parameters should I pay attention to in
> > order to
> > > scale the
> > > > > server to
> > > > > > serve 10,000 or more connections without
> > burning
> > > up CPU?
> > > > > >
> > > > > > The old server can serve 12000
> connections
> > with 4
> > > > instance at
> > > > > <= 50%
> > > > > > CPU usage.
> > > > > >
> > > > > > Would it be better to run 2 instances of
> > NIO server
> > > > instead
> > > > > of 4?
> > > > > >
> > > > > > Your opinion and suggestion is greatly
> > appreciated.
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > > Fay
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > < http://java.sun.com/docs/performance/
> > > > <http://java.sun.com/docs/performance/
> > > < http://java.sun.com/docs/performance/>>
> > > > > <http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>> <mailto:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > <http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/>>
> > > >
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > >
> > > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > < http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > For additional commands, e-mail:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > >
> > >
> >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > <http://java.sun.com/docs/performance/>
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > For additional commands, e-mail:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> >
> >
>
>
> --
>
> Charlie Hunt
> Java Performance Engineer
> 630.285.7708 x47708 (Internal)
>
> <http://java.sun.com/docs/performance/>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
>
>

-- 
Charlie Hunt
Java Performance Engineer
630.285.7708 x47708 (Internal)
<http://java.sun.com/docs/performance/>