Also just realized that we currently do not set tcpNoDelay on the
"Connector / client" side.
I will open a bug for this issue so that it is set the same by default
as the "SelectorHandler / server" side of the Socket.
charlie ...
charlie hunt wrote:
> This is good news!
>
> Glad to hear you have found a resolution.
>
> You will probably need to explicitly make a call to
> TCPSelectorHandler.setTcpNoDelay(false) when you configure Grizzly.
>
> I will add a comment to the code and JavaDoc for TCPSelectorHandler
> which reflects your findings.
>
> charlie ...
>
>
> Fay Zheng wrote:
>> It seems that the fragmented packet behavior goes away after I set
>> tcpNoDelay to false explicitly in the socket connection settings for
>> both client and server.
>>
>> thanks,
>> Fay
>>
>>
>> On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>> wrote:
>>
>> Any chance you can try your configuration on a platform other than
>> Linux
>> such as OpenSolaris (x86) or Windows ?
>>
>> Linux has a history of being a little trouble-some when it comes to
>> working well with Java NIO.
>>
>> This is the first I have heard of someone observing the sort of
>> thing
>> you are describing where you read small portions of data at a time.
>>
>> Can you isolate your testing system completely independent of all
>> other
>> LAN traffic, (i.e. isolate the test system to include only the
>> machine
>> your are testing on a load machine) ?
>>
>> charlie ...
>>
>> Fay Zheng wrote:
>> > Disable TcpNoDelay does not seem to help. We still receive tons of
>> > fragmented packets on single read.
>> >
>> > Has anyone had similar experience before?
>> >
>> > thanks,
>> > Fay
>> >
>> >
>> > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>
>> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
>> wrote:
>> >
>> > Hi Fay,
>> >
>> > You could try it. It may help.
>> >
>> > If it does indeed improve things for you, let us know.
>> >
>> > charlie ...
>> >
>> > Fay Zheng wrote:
>> > > Thank you for your help. I also noticed that there are
>> some TCP
>> > > specific behaviors. Looks like tcp connection is sending 1
>> byte
>> > > (partial data) on one read, then the rest of the request
>> content on
>> > > next read. So we have to assembling the request. This
>> happens quiet
>> > > often during our load test. Should I enable " tcpNoDelay"?
>> Will it
>> > > help to turn off the behavior?
>> > >
>> > > thanks,
>> > > Fay
>> > >
>> > >
>> > > On 8/14/07, *charlie hunt* <charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>
>> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
>> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>> wrote:
>> > >
>> > > There's a couple places in the grizzly source code you
>> would
>> > need to
>> > > explicitly modify from that I looked at yesterday.
>> > >
>> > > Here's what I identified:
>> > >
>> > > 1.) In Controller.java , doSelect() method. Remove this
>> > logic towards
>> > > the end of the method:
>> > > iterator = readyKeys.iterator();
>> > > long currentTime =
>> System.currentTimeMillis ();
>> > > while (iterator.hasNext() &&
>> > selectorHandler.isOpen()) {
>> > > key = iterator.next();
>> > > selectionKeyHandler.expire
>> (key,currentTime);
>> > > }
>> > > *Note: In the trunk, Alexy just moved this logic to the
>> > > DefaultSelectionKeyHandler.
>> > >
>> > > 2.) If you are using the TCPSelectorHandler, in
>> onReadOps(),
>> > you don't
>> > > need the currentTime to initialized, you can just set
>> it to
>> > 0, or
>> > > remove
>> > > the currentTime and replace currentTime with 0L in the
>> > > selectionKeyHandler.register(key, currentTime)
>> call. Same thing
>> > > applies
>> > > to onWriteOps(). In onAcceptInterest(), you can
>> replace the
>> > call to
>> > > System.currentTimeMillis () with 0L.
>> > >
>> > > 3.) In DefaultSelectionKeyHandler.expire(), just make
>> the entire
>> > > method
>> > > a no-op. Just tell it to return as soon as it is
>> entered.
>> > >
>> > > I think that is it. That is all I could find.
>> > >
>> > > charlie ...
>> > >
>> > > Fay Zheng wrote:
>> > > > Thank you! We've already override the expire method
>> in the
>> > > > DefaultSelectionKeyHandler to do nothing for our
>> application:
>> > > >
>> > > > DefaultSelectionKeyHandler keyHandler =
>> > > >
>> > > > *new* DefaultSelectionKeyHandler() { *public* *void*
>> > > > expire(SelectionKey key) { *return*;
>> > > >
>> > > > }
>> > > >
>> > > > };
>> > > >
>> > > > However this approach looks quiet expensive in the
>> profiling
>> > > data. For
>> > > > now I have to customize the grizzly source code to
>> remove the
>> > > expiring
>> > > > of key part.
>> > > >
>> > > > thanks,
>> > > > Fay
>> > > >
>> > > >
>> > > > On 8/13/07, *charlie hunt* < charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>
>> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> > > <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>>>
>> > > > <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>>
>> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
>> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>
>> wrote:
>> > > >
>> > > > Hi Fay,
>> > > >
>> > > > At the moment, there is not a way to turn off
>> expiring
>> > of a
>> > > key.
>> > > >
>> > > > But, it is functionality we should have and I
>> know of
>> > > another project
>> > > > which will want this functionality disabled as
>> well.
>> > > >
>> > > > I will file an enhancement issue to add this
>> > functionality.
>> > > >
>> > > > charlie ...
>> > > >
>> > > > Fay Zheng wrote:
>> > > > > Is there a way to turn off expire of key,
>> since it
>> > is not
>> > > > applicable
>> > > > > to persistent connections:
>> > > > >
>> > > > > iterator = readyKeys.iterator();
>> > > > >
>> > > > > while (iterator.hasNext() &&
>> > selectorHandler.isOpen()) {
>> > > > >
>> > > > > key = iterator.next();
>> > > > >
>> > > > > selectionKeyHandler.expire(key);
>> > > > >
>> > > > >
>> > > > >
>> > > > > thanks,
>> > > > >
>> > > > > Fay
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On 8/10/07, *Fay Zheng* < fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>
>> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
>> > > <mailto:fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>>>
>> > > > <mailto: fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>
>> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
>> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
>> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>
>> > > > > <mailto: fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>
>> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
>> > > <mailto: fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>>>
>> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
>> <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
>> > > <mailto: fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>
>> > <mailto:fzheng1998_at_gmail.com
>> <mailto:fzheng1998_at_gmail.com>>>>>> wrote:
>> > > > >
>> > > > > Here's the command to run the application:
>> > > > >
>> > > > > java -server -verbose:gc -XX:+PrintGC
>> > > > > Details -XX:+PrintGCTimeStamps -
>> > > > > Dcom.pg.env=/props/env/dev/pg/trx.properties
>> > > > > -Dcom.sun.management.jmxremote
>> > > > > - Dcom.sun.management.jmxremote.port=9991
>> > > > >
>> -Dcom.sun.management.jmxremote.authenticate=false
>> > > > > -Dcom.sun.management.jmxremote.ssl=false
>> > > > > - Duser.region=US -Xmx700m
>> com.pg.serv.room.Main
>> > > > >
>> > > > > The GC is almost 10% for NIO, 5% for none
>> NIO
>> > server.
>> > > > >
>> > > > > thansk,
>> > > > > Fay
>> > > > >
>> > > > > On 8/8/07, *charlie hunt* <
>> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
>> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> > > <mailto: charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>>>
>> > > > <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>
>> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
>> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
>> > > > > <mailto: charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>
>> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> > > <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>>>
>> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
>> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
>> > > <mailto:charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
>> <mailto:charlie.hunt_at_sun.com>>>>>>
>> > > > wrote:
>> > > > >
>> > > > > In addition to "RG's" suggestion, I
>> would:
>> > > > >
>> > > > > 1.) Set use_direct_buffer =
>> false (Grizzly
>> > will use
>> > > > > HeapByteBuffer instead)
>> > > > > 2.) You might also reduce the number of
>> > instances
>> > > of the
>> > > > > server down to
>> > > > > 2 or even 1. NIO should give you much
>> better
>> > > > scalability and
>> > > > > should
>> > > > > have no problems scaling to 12,000
>> connections.
>> > > > > 3.) If you can, I would also go to the
>> > latest JDK 6_02
>> > > > version.
>> > > > >
>> > > > > We might also need to do some further
>> fine
>> > tuning
>> > > of the
>> > > > JVM too.
>> > > > >
>> > > > > Could you share with us the full java
>> > command line
>> > > args you
>> > > > > are using?
>> > > > > It might also be useful to see the
>> output from
>> > > > > -XX:+PrintGCDetails when
>> > > > > you are running with NIO.
>> > > > >
>> > > > > hths,
>> > > > >
>> > > > > charlie ...
>> > > > >
>> > > > > Fay Zheng wrote:
>> > > > > > We add Grizzly Nio engine to one of
>> existing
>> > > server,
>> > > > which
>> > > > > used to be
>> > > > > > one thread per connection classic
>> model.
>> > During load
>> > > > test we
>> > > > > noticed
>> > > > > > that NIO server consumes twice CPU
>> power than
>> > > the classic
>> > > > > model when
>> > > > > > serving the same amount of
>> connections.
>> > > > > >
>> > > > > > Here's our configurations:
>> > > > > >
>> > > > > > Here are some statistics:
>> > > > > >
>> > > > > > Hardware: Dells 2950 with 16 GB RAM,
>> total 8
>> > > > processors, Intel(R)
>> > > > > > Xeon(R) CPU E5345 @ 2.33GHz , Linux
>> > kernel 2.6
>> > > > > >
>> > > > > > Software: JDK v1.5.6
>> > > > > >
>> > > > > > There are 4 instance of the server
>> application
>> > > > configured on
>> > > > > one box
>> > > > > >
>> > > > > > Grizzly Configs:
>> > > > > >
>> > > > > > 100 max worker threads (8K buffer
>> size)
>> > > > > >
>> > > > > > 150 max output buffer pool size (4K
>> buffer
>> > size)
>> > > > > >
>> > > > > > use_direct_buffer = true
>> > > > > >
>> > > > > > selector timeout = 500
>> > > > > >
>> > > > > > Keep alive is true (persistent
>> connections)
>> > > > > >
>> > > > > >
>> > > > > > Tested with : 7200 client connections
>> > > > > >
>> > > > > > Total CPU usage for all 4 instances
>> are
>> > 44.59% (with
>> > > > NIO engine)
>> > > > > >
>> > > > > > CPU usage is 24.76% (without Nio
>> engine)
>> > > > > >
>> > > > > > So far as memory, NIO server uses
>> slightly
>> > less
>> > > memory in
>> > > > > this case.
>> > > > > >
>> > > > > >
>> > > > > > The memory consumption could be linear
>> > because
>> > > we keep
>> > > > lots
>> > > > > of state
>> > > > > > information for each connection
>> (user).
>> > Other than
>> > > > that, what
>> > > > > other
>> > > > > > parameters should I pay attention
>> to in
>> > order to
>> > > scale the
>> > > > > server to
>> > > > > > serve 10,000 or more connections
>> without
>> > burning
>> > > up CPU?
>> > > > > >
>> > > > > > The old server can serve 12000
>> connections
>> > with 4
>> > > > instance at
>> > > > > <= 50%
>> > > > > > CPU usage.
>> > > > > >
>> > > > > > Would it be better to run 2
>> instances of
>> > NIO server
>> > > > instead
>> > > > > of 4?
>> > > > > >
>> > > > > > Your opinion and suggestion is greatly
>> > appreciated.
>> > > > > >
>> > > > > >
>> > > > > > thanks,
>> > > > > > Fay
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Charlie Hunt
>> > > > > Java Performance Engineer
>> > > > > 630.285.7708 x47708 (Internal)
>> > > > >
>> > > > > < http://java.sun.com/docs/performance/
>> > > > <http://java.sun.com/docs/performance/
>> > > < http://java.sun.com/docs/performance/>>
>> > > > > <http://java.sun.com/docs/performance/
>> <http://java.sun.com/docs/performance/>>>
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail:
>> > > > dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
>> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
>> > > > > <mailto:
>> > dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
>> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
>> > > > > For additional commands, e-mail:
>> > > > dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>> <mailto:
>> > > dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>>
>> > > > > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>
>> > > > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > >
>> > > > Charlie Hunt
>> > > > Java Performance Engineer
>> > > > 630.285.7708 x47708 (Internal)
>> > > >
>> > > > <http://java.sun.com/docs/performance/
>> > > > < http://java.sun.com/docs/performance/>>
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail:
>> > dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
>> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
>> > > > For additional commands, e-mail:
>> > > dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>
>> > > > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>>
>> > > >
>> > > >
>> > >
>> > > --
>> > >
>> > > Charlie Hunt
>> > > Java Performance Engineer
>> > > 630.285.7708 x47708 (Internal)
>> > >
>> > > < http://java.sun.com/docs/performance/
>> <http://java.sun.com/docs/performance/>>
>> > >
>> > >
>> > >
>> >
>> ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail:
>> dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
>> > > For additional commands, e-mail:
>> > dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> > > <mailto: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>>
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > Charlie Hunt
>> > Java Performance Engineer
>> > 630.285.7708 x47708 (Internal)
>> >
>> > <http://java.sun.com/docs/performance/>
>> >
>> >
>> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
>> > For additional commands, e-mail:
>> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
>> > <mailto:dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>>
>> >
>> >
>>
>>
>> --
>>
>> Charlie Hunt
>> Java Performance Engineer
>> 630.285.7708 x47708 (Internal)
>>
>> <http://java.sun.com/docs/performance/>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>> <mailto:dev-help_at_grizzly.dev.java.net>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>
--
Charlie Hunt
Java Performance Engineer
630.285.7708 x47708 (Internal)
<http://java.sun.com/docs/performance/>