Hi Fay,
Are you most interested in "raw throughput" with this application with
little or no interest in "pause time / responsiveness" ?
Or, is "pause time / responsiveness" more important than throughput ?
Do you have a threshold at which you want all your GC pause times to be
less than?
Let's start with that ... then we'll work on tuning gc for your NIO server.
charlie ...
Fay Zheng wrote:
> Now we have a GC problem. The NIO server is using a lot more GC time:
>
> 1) GC data for NIO server :
>
>
> GC collector: Name = 'PS Scavenge', Collections = 737, Total time
> spent = 2 minutes
>
> GC collector: Name = 'PS MarkSweep', Collections = 59, Total time
> spent = 16.414 seconds
>
> Committed memory: 97, 088 kbytes
>
>
>
> 1) GC data for none - NIO server :
>
> GC collector: Name = 'PS Scavenge', Collections = 1075, Total time
> spent = 14.349 seconds
>
> GC collector: Name = 'PS MarkSweep', Collections = 17, Total time
> spent = 6.511 seconds
>
> Committed memory: 102,464 kbytes
>
> And we are using the default GC settings from JVM 1.5.12.
>
> Any ideas on how to configure the GC for NIO engine will be greatly
> appreciated?
>
> thanks,
> Fay
>
> On 8/20/07, *Fay Zheng* <fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>> wrote:
>
> It seems that the fragmented packet behavior goes away after I set
> tcpNoDelay to false explicitly in the socket connection
> settings for both client and server.
>
> thanks,
> Fay
>
>
> On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> wrote:
>
> Any chance you can try your configuration on a platform other
> than Linux
> such as OpenSolaris (x86) or Windows ?
>
> Linux has a history of being a little trouble-some when it
> comes to
> working well with Java NIO.
>
> This is the first I have heard of someone observing the sort
> of thing
> you are describing where you read small portions of data at a
> time.
>
> Can you isolate your testing system completely independent of
> all other
> LAN traffic, (i.e. isolate the test system to include only the
> machine
> your are testing on a load machine) ?
>
> charlie ...
>
> Fay Zheng wrote:
> > Disable TcpNoDelay does not seem to help. We still receive
> tons of
> > fragmented packets on single read.
> >
> > Has anyone had similar experience before?
> >
> > thanks,
> > Fay
> >
> >
> > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> wrote:
> >
> > Hi Fay,
> >
> > You could try it. It may help.
> >
> > If it does indeed improve things for you, let us know.
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Thank you for your help. I also noticed that there are
> some TCP
> > > specific behaviors. Looks like tcp connection is
> sending 1 byte
> > > (partial data) on one read, then the rest of the
> request content on
> > > next read. So we have to assembling the request. This
> happens quiet
> > > often during our load test. Should I enable "
> tcpNoDelay"? Will it
> > > help to turn off the behavior?
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/14/07, *charlie hunt* < charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>> wrote:
> > >
> > > There's a couple places in the grizzly source code
> you would
> > need to
> > > explicitly modify from that I looked at yesterday.
> > >
> > > Here's what I identified:
> > >
> > > 1.) In Controller.java , doSelect() method. Remove
> this
> > logic towards
> > > the end of the method:
> > > iterator = readyKeys.iterator();
> > > long currentTime =
> System.currentTimeMillis ();
> > > while (iterator.hasNext() &&
> > selectorHandler.isOpen()) {
> > > key = iterator.next();
> > > selectionKeyHandler.expire
> (key,currentTime);
> > > }
> > > *Note: In the trunk, Alexy just moved this logic to the
> > > DefaultSelectionKeyHandler.
> > >
> > > 2.) If you are using the TCPSelectorHandler, in
> onReadOps(),
> > you don't
> > > need the currentTime to initialized, you can just
> set it to
> > 0, or
> > > remove
> > > the currentTime and replace currentTime with 0L in the
> > > selectionKeyHandler.register(key, currentTime)
> call. Same thing
> > > applies
> > > to onWriteOps(). In onAcceptInterest(), you can
> replace the
> > call to
> > > System.currentTimeMillis () with 0L.
> > >
> > > 3.) In DefaultSelectionKeyHandler.expire(), just
> make the entire
> > > method
> > > a no-op. Just tell it to return as soon as it is
> entered.
> > >
> > > I think that is it. That is all I could find.
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Thank you! We've already override the expire
> method in the
> > > > DefaultSelectionKeyHandler to do nothing for our
> application:
> > > >
> > > > DefaultSelectionKeyHandler keyHandler =
> > > >
> > > > *new* DefaultSelectionKeyHandler() { *public* *void*
> > > > expire(SelectionKey key) { *return*;
> > > >
> > > > }
> > > >
> > > > };
> > > >
> > > > However this approach looks quiet expensive in
> the profiling
> > > data. For
> > > > now I have to customize the grizzly source code
> to remove the
> > > expiring
> > > > of key part.
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/13/07, *charlie hunt* < charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>> wrote:
> > > >
> > > > Hi Fay,
> > > >
> > > > At the moment, there is not a way to turn off
> expiring
> > of a
> > > key.
> > > >
> > > > But, it is functionality we should have and I
> know of
> > > another project
> > > > which will want this functionality disabled
> as well.
> > > >
> > > > I will file an enhancement issue to add this
> > functionality.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Is there a way to turn off expire of key,
> since it
> > is not
> > > > applicable
> > > > > to persistent connections:
> > > > >
> > > > > iterator = readyKeys.iterator ();
> > > > >
> > > > > while (iterator.hasNext() &&
> > selectorHandler.isOpen()) {
> > > > >
> > > > > key = iterator.next();
> > > > >
> > > > > selectionKeyHandler.expire(key);
> > > > >
> > > > >
> > > > >
> > > > > thanks,
> > > > >
> > > > > Fay
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 8/10/07, *Fay Zheng* <
> fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>>> wrote:
> > > > >
> > > > > Here's the command to run the application:
> > > > >
> > > > > java -server -verbose:gc -XX:+PrintGC
> > > > > Details -XX:+PrintGCTimeStamps -
> > > > >
> Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > -Dcom.sun.management.jmxremote
> > > > > - Dcom.sun.management.jmxremote.port=9991
> > > > >
> -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > > - Duser.region=US -Xmx700m
> com.pg.serv.room.Main
> > > > >
> > > > > The GC is almost 10% for NIO, 5% for
> none NIO
> > server.
> > > > >
> > > > > thansk,
> > > > > Fay
> > > > >
> > > > > On 8/8/07, *charlie hunt* <
> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>>
> > > > wrote:
> > > > >
> > > > > In addition to "RG's" suggestion, I
> would:
> > > > >
> > > > > 1.) Set use_direct_buffer =
> false (Grizzly
> > will use
> > > > > HeapByteBuffer instead)
> > > > > 2.) You might also reduce the
> number of
> > instances
> > > of the
> > > > > server down to
> > > > > 2 or even 1. NIO should give you
> much better
> > > > scalability and
> > > > > should
> > > > > have no problems scaling to 12,000
> connections.
> > > > > 3.) If you can, I would also go to the
> > latest JDK 6_02
> > > > version.
> > > > >
> > > > > We might also need to do some
> further fine
> > tuning
> > > of the
> > > > JVM too.
> > > > >
> > > > > Could you share with us the full java
> > command line
> > > args you
> > > > > are using?
> > > > > It might also be useful to see the
> output from
> > > > > -XX:+PrintGCDetails when
> > > > > you are running with NIO.
> > > > >
> > > > > hths,
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > We add Grizzly Nio engine to one
> of existing
> > > server,
> > > > which
> > > > > used to be
> > > > > > one thread per connection classic
> model.
> > During load
> > > > test we
> > > > > noticed
> > > > > > that NIO server consumes twice
> CPU power than
> > > the classic
> > > > > model when
> > > > > > serving the same amount of
> connections.
> > > > > >
> > > > > > Here's our configurations:
> > > > > >
> > > > > > Here are some statistics:
> > > > > >
> > > > > > Hardware: Dells 2950 with 16 GB
> RAM, total 8
> > > > processors, Intel(R)
> > > > > > Xeon(R) CPU E5345 @ 2.33GHz , Linux
> > kernel 2.6
> > > > > >
> > > > > > Software: JDK v1.5.6
> > > > > >
> > > > > > There are 4 instance of the
> server application
> > > > configured on
> > > > > one box
> > > > > >
> > > > > > Grizzly Configs:
> > > > > >
> > > > > > 100 max worker threads (8K buffer
> size)
> > > > > >
> > > > > > 150 max output buffer pool size
> (4K buffer
> > size)
> > > > > >
> > > > > > use_direct_buffer = true
> > > > > >
> > > > > > selector timeout = 500
> > > > > >
> > > > > > Keep alive is true (persistent
> connections)
> > > > > >
> > > > > >
> > > > > > Tested with : 7200 client connections
> > > > > >
> > > > > > Total CPU usage for all 4
> instances are
> > 44.59% (with
> > > > NIO engine)
> > > > > >
> > > > > > CPU usage is 24.76% (without Nio
> engine)
> > > > > >
> > > > > > So far as memory, NIO server uses
> slightly
> > less
> > > memory in
> > > > > this case.
> > > > > >
> > > > > >
> > > > > > The memory consumption could be
> linear
> > because
> > > we keep
> > > > lots
> > > > > of state
> > > > > > information for each connection
> (user).
> > Other than
> > > > that, what
> > > > > other
> > > > > > parameters should I pay attention
> to in
> > order to
> > > scale the
> > > > > server to
> > > > > > serve 10,000 or more connections
> without
> > burning
> > > up CPU?
> > > > > >
> > > > > > The old server can serve 12000
> connections
> > with 4
> > > > instance at
> > > > > <= 50%
> > > > > > CPU usage.
> > > > > >
> > > > > > Would it be better to run 2
> instances of
> > NIO server
> > > > instead
> > > > > of 4?
> > > > > >
> > > > > > Your opinion and suggestion is
> greatly
> > appreciated.
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > > Fay
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > < http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/
> > > < http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>>
> > > > >
> <http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
>
> > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>> <mailto:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > < http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>>
> > > >
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
>
> > > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > >
> > > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > < http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > For additional commands, e-mail:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net> <mailto:
> dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > >
> > >
> >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > < http://java.sun.com/docs/performance/>
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > For additional commands, e-mail:
> dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> >
> >
>
>
> --
>
> Charlie Hunt
> Java Performance Engineer
> 630.285.7708 x47708 (Internal)
>
> <http://java.sun.com/docs/performance/>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
>
>
>
--
Charlie Hunt
Java Performance Engineer
630.285.7708 x47708 (Internal)
<http://java.sun.com/docs/performance/>