Hi Fay,
Are you running anything other than the 4 server instances on these
machines ? At a 700M Java heap, that's only about 2.8G used for Java
heap. If we do some rather conservative calculations ;-)
1 GiB for OS, 300M addl overhead per JVM + 700M Java heap ~ 1GiB per
server instance then we're only using about 5 GiB of that total 16GiB.
We can likely use a larger Java heap.
Let's modify your Java command line as follows:
java -server \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps \
-Xmx1400m -Xms1400m -Xmn200m -Xss128k \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-Dcom.pg.props=/props/serv/room/trex1n.properties \
-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.EPollSelectorProvider \
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9991 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Duser.region=US \
com.pg.serv.room.Main |& logfile room-p21 200000) >& /dev/null &
This will switch over to using the concurrent old generation collector
along a parallel young generation collector. The idea here is to use
the concurrent collector to (hopefully) avoid full gcs and keep pause
times minimal.
Let's use this as a starting point.
If you can collect the gc logs from this setup and send them to me, I
can look at them see if we can do some further fine tuning.
We may need to do some fine tuning if we find minor gc's are taking too
long or the concurrent collector is not keeping up to the rate at which
objects are getting promoted from young gen to old gen.
charlie ...
Fay Zheng wrote:
> We use the same hardware for our test as well as production machine:
> Hardware: Dells 2950 with 16 GB RAM, two Quad-Core Intel(R) Xeon(R)
> E5345 @ 2.33GHz, Linux kernel 2.6.
> 4 server instances running on one box
> JDK v1.5.12 with epoll enabled
> heap size is set to 700M for each JVM.
>
> We also have some log rotation and monitoring process running on the
> same box. They will wake up from time to time.
>
> Here's the command to start one instance:
> java -server -verbose:gc -XX:+PrintGC
> Details -XX:+PrintGCTimeStamps -Dcom.pg.props=/props/s
> erv/room/trex1n.properties
> -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.EPo
> llSelectorProvider -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote
> .port=9991 -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.managemen
> t.jmxremote.ssl=false -Duser.region=US -Xmx700m -Xss128k
> com.pg.serv.room.Main
> |& logfile room-p21 200000) >& /dev/null &
>
>
>
> On 8/21/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> wrote:
>
> Sounds like we might be best to work with the concurrent collector.
>
> How much physical memory do you have on your test machine (and your
> production machine) ?
>
> And, do you have any additional applications you are wanting to run on
> the same machine at the same time ?
>
> IIRC, I think you were using a 700M java heap?
>
> Can you send your entire java command line?
>
> We'll likely configure the JVM with a rather small young
> generation with
> a large old generation with the premise of promoting "medium lived"
> object into old generation and letting the concurrent collector clean
> these up. The challenge will be sizing old generation such that we
> don't over-run it with new garbage too fast that the concurrent
> collector can't free it, (I refer to this as "losing the race".).
>
> charlie ...
>
>
> Fay Zheng wrote:
> > Yes. We care more about the raw throughput. Our main goal is to
> > achieve higher maximum capacity through this project. Most of our
> > traffic is small keepalive messages, which has a timeout of 30
> > seconds. However the desirable response time for game specific
> > messages is no more than 500 milliseconds.
> >
> > I looked at the GC log in more detail, the full GC looks better in
> > NIO, but we have a much worse minor GC time for YoungGen with NIO
> > engine when compare to old classic gameserver:
> >
> >
> > rserv_log_070817_125402:7436.600: [Full GC [PSYoungGen:
> > 5488K->0K(61632K)] [PSOldGen: 41294K->35274K(44992K)]
> > 46782K->35274K(106624K) [PSPermGen: 16366K->16366K(26752K)],
> 0.1906040
> > secs]
> >
> > rserv_log_070817_125402:7442.548: [GC [PSYoungGen:
> > 45568K->1504K(61504K)] 80842K->36778K(106496K), 0.0078060 secs]
> >
> > rserv_log_070817_125402:7448.310: [GC [PSYoungGen:
> > 47072K->2112K(63168K)] 82346K->38638K(108160K), 0.0838490 secs]
> >
> > rserv_log_070817_125402:7449.229: [GC [PSYoungGen:
> > 49856K->5912K(53696K)] 86382K->44402K(98688K), 0.6270710 secs]
> >
> > rserv_log_070817_125402:7450.578: [GC [PSYoungGen:
> > 53656K->5856K(64192K)] 92146K->50203K(109184K), 1.0315760 secs]
> >
> > rserv_log_070817_125402:7451.609: [Full GC [PSYoungGen:
> > 5856K->0K(64192K)] [PSOldGen: 44347K->38032K(48896K)]
> > 50203K->38032K(113088K) [PSPermGen: 16374K->16374K(26240K)],
> 0.2052210
> > secs]
> >
> >
> >
> > There are quiet some minor GCs in NIO is taking more than one
> second.
> > The average minor GC time is 0.206815. While the average time of
> minor
> > GC for none-NIO server is 0.0133541.
> >
> > Full_GC_baseline avg / max duration in seconds: 0.39 / 0.58
> >
> >
> >
> > Full_GC_NIO avg / max duration in seconds: 0.28 / 0.41
> >
> > thanks,
> >
> > Fay
> >
> >
> >
> > On 8/21/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>> wrote:
> >
> > Hi Fay,
> >
> > Are you most interested in "raw throughput" with this
> application with
> > little or no interest in "pause time / responsiveness" ?
> >
> > Or, is "pause time / responsiveness" more important than
> throughput ?
> >
> > Do you have a threshold at which you want all your GC pause
> times
> > to be
> > less than?
> >
> > Let's start with that ... then we'll work on tuning gc for your
> > NIO server.
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Now we have a GC problem. The NIO server is using a lot
> more GC
> > time:
> > >
> > > 1) GC data for NIO server :
> > >
> > >
> > > GC collector: Name = 'PS Scavenge', Collections = 737,
> Total time
> > > spent = 2 minutes
> > >
> > > GC collector: Name = 'PS MarkSweep', Collections = 59,
> Total time
> > > spent = 16.414 seconds
> > >
> > > Committed memory: 97, 088 kbytes
> > >
> > >
> > >
> > > 1) GC data for none - NIO server :
> > >
> > > GC collector: Name = 'PS Scavenge', Collections = 1075,
> Total time
> > > spent = 14.349 seconds
> > >
> > > GC collector: Name = 'PS MarkSweep', Collections = 17,
> Total time
> > > spent = 6.511 seconds
> > >
> > > Committed memory: 102,464 kbytes
> > >
> > > And we are using the default GC settings from JVM 1.5.12.
> > >
> > > Any ideas on how to configure the GC for NIO engine will
> be greatly
> > > appreciated?
> > >
> > > thanks,
> > > Fay
> > >
> > > On 8/20/07, *Fay Zheng* <fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>> wrote:
> > >
> > > It seems that the fragmented packet behavior goes away
> after
> > I set
> > > tcpNoDelay to false explicitly in the socket connection
> > > settings for both client and server.
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>> wrote:
> > >
> > > Any chance you can try your configuration on a
> platform
> > other
> > > than Linux
> > > such as OpenSolaris (x86) or Windows ?
> > >
> > > Linux has a history of being a little trouble-some
> when it
> > > comes to
> > > working well with Java NIO.
> > >
> > > This is the first I have heard of someone
> observing the sort
> > > of thing
> > > you are describing where you read small portions
> of data
> > at a
> > > time.
> > >
> > > Can you isolate your testing system completely
> > independent of
> > > all other
> > > LAN traffic, (i.e. isolate the test system to include
> > only the
> > > machine
> > > your are testing on a load machine) ?
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Disable TcpNoDelay does not seem to help. We
> still receive
> > > tons of
> > > > fragmented packets on single read.
> > > >
> > > > Has anyone had similar experience before?
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com> >
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>
> > > wrote:
> > > >
> > > > Hi Fay,
> > > >
> > > > You could try it. It may help.
> > > >
> > > > If it does indeed improve things for you,
> let us know.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Thank you for your help. I also noticed that
> > there are
> > > some TCP
> > > > > specific behaviors. Looks like tcp
> connection is
> > > sending 1 byte
> > > > > (partial data) on one read, then the rest
> of the
> > > request content on
> > > > > next read. So we have to assembling the
> request.
> > This
> > > happens quiet
> > > > > often during our load test. Should I enable "
> > > tcpNoDelay"? Will it
> > > > > help to turn off the behavior?
> > > > >
> > > > > thanks,
> > > > > Fay
> > > > >
> > > > >
> > > > > On 8/14/07, *charlie hunt* <
> > charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>> wrote:
> > > > >
> > > > > There's a couple places in the grizzly
> > source code
> > > you would
> > > > need to
> > > > > explicitly modify from that I looked at
> > yesterday.
> > > > >
> > > > > Here's what I identified:
> > > > >
> > > > > 1.) In Controller.java , doSelect()
> > method. Remove
> > > this
> > > > logic towards
> > > > > the end of the method:
> > > > > iterator =
> readyKeys.iterator();
> > > > > long currentTime =
> > > System.currentTimeMillis ();
> > > > > while (iterator.hasNext() &&
> > > > selectorHandler.isOpen()) {
> > > > > key = iterator.next();
> > > > > selectionKeyHandler.expire
> > > (key,currentTime);
> > > > > }
> > > > > *Note: In the trunk, Alexy just moved
> this
> > logic to the
> > > > > DefaultSelectionKeyHandler.
> > > > >
> > > > > 2.) If you are using the
> TCPSelectorHandler, in
> > > onReadOps(),
> > > > you don't
> > > > > need the currentTime to initialized,
> you can
> > just
> > > set it to
> > > > 0, or
> > > > > remove
> > > > > the currentTime and replace
> currentTime with
> > 0L in the
> > > > > selectionKeyHandler.register(key,
> currentTime)
> > > call. Same thing
> > > > > applies
> > > > > to onWriteOps(). In
> onAcceptInterest(), you
> > can
> > > replace the
> > > > call to
> > > > > System.currentTimeMillis () with 0L.
> > > > >
> > > > > 3.) In
> DefaultSelectionKeyHandler.expire (),
> > just
> > > make the entire
> > > > > method
> > > > > a no-op. Just tell it to return as
> soon as it is
> > > entered.
> > > > >
> > > > > I think that is it. That is all I
> could find.
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > Thank you! We've already override
> the expire
> > > method in the
> > > > > > DefaultSelectionKeyHandler to do nothing
> > for our
> > > application:
> > > > > >
> > > > > > DefaultSelectionKeyHandler keyHandler =
> > > > > >
> > > > > > *new* DefaultSelectionKeyHandler() {
> > *public* *void*
> > > > > > expire(SelectionKey key) { *return*;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > };
> > > > > >
> > > > > > However this approach looks quiet
> expensive in
> > > the profiling
> > > > > data. For
> > > > > > now I have to customize the grizzly
> source
> > code
> > > to remove the
> > > > > expiring
> > > > > > of key part.
> > > > > >
> > > > > > thanks,
> > > > > > Fay
> > > > > >
> > > > > >
> > > > > > On 8/13/07, *charlie hunt* <
> > charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>
> > > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>>> wrote:
> > > > > >
> > > > > > Hi Fay,
> > > > > >
> > > > > > At the moment, there is not a
> way to
> > turn off
> > > expiring
> > > > of a
> > > > > key.
> > > > > >
> > > > > > But, it is functionality we should
> > have and I
> > > know of
> > > > > another project
> > > > > > which will want this functionality
> > disabled
> > > as well.
> > > > > >
> > > > > > I will file an enhancement issue to
> > add this
> > > > functionality.
> > > > > >
> > > > > > charlie ...
> > > > > >
> > > > > > Fay Zheng wrote:
> > > > > > > Is there a way to turn off
> expire of
> > key,
> > > since it
> > > > is not
> > > > > > applicable
> > > > > > > to persistent connections:
> > > > > > >
> > > > > > > iterator = readyKeys.iterator ();
> > > > > > >
> > > > > > > while ( iterator.hasNext() &&
> > > > selectorHandler.isOpen()) {
> > > > > > >
> > > > > > > key = iterator.next();
> > > > > > >
> > > > > > > selectionKeyHandler.expire(key);
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > Fay
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 8/10/07, *Fay Zheng* <
> > > fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>
> <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>>
> > > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>>>
> > > > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>
> <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>>>>> wrote:
> > > > > > >
> > > > > > > Here's the command to run the
> > application:
> > > > > > >
> > > > > > > java -server -verbose:gc
> > -XX:+PrintGC
> > > > > > > Details
> -XX:+PrintGCTimeStamps -
> > > > > > >
> > > Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > > > -
> Dcom.sun.management.jmxremote
> > > > > > > -
> > Dcom.sun.management.jmxremote.port=9991
> > > > > > >
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > >
> > -Dcom.sun.management.jmxremote.ssl=false
> > > > > > > - Duser.region=US -Xmx700m
> > > com.pg.serv.room.Main
> > > > > > >
> > > > > > > The GC is almost 10% for
> NIO, 5% for
> > > none NIO
> > > > server.
> > > > > > >
> > > > > > > thansk,
> > > > > > > Fay
> > > > > > >
> > > > > > > On 8/8/07, *charlie hunt* <
> > > charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>
> > > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>>
> > > > > > > <mailto:
> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>>>>
> > > > > > wrote:
> > > > > > >
> > > > > > > In addition to "RG's"
> > suggestion, I
> > > would:
> > > > > > >
> > > > > > > 1.) Set
> use_direct_buffer =
> > > false (Grizzly
> > > > will use
> > > > > > > HeapByteBuffer instead)
> > > > > > > 2.) You might also
> reduce the
> > > number of
> > > > instances
> > > > > of the
> > > > > > > server down to
> > > > > > > 2 or even 1. NIO should
> > give you
> > > much better
> > > > > > scalability and
> > > > > > > should
> > > > > > > have no problems
> scaling to
> > 12,000
> > > connections.
> > > > > > > 3.) If you can, I
> would also
> > go to the
> > > > latest JDK 6_02
> > > > > > version.
> > > > > > >
> > > > > > > We might also need to
> do some
> > > further fine
> > > > tuning
> > > > > of the
> > > > > > JVM too.
> > > > > > >
> > > > > > > Could you share with
> us the
> > full java
> > > > command line
> > > > > args you
> > > > > > > are using?
> > > > > > > It might also be
> useful to
> > see the
> > > output from
> > > > > > > -XX:+PrintGCDetails when
> > > > > > > you are running with NIO.
> > > > > > >
> > > > > > > hths,
> > > > > > >
> > > > > > > charlie ...
> > > > > > >
> > > > > > > Fay Zheng wrote:
> > > > > > > > We add Grizzly Nio
> engine
> > to one
> > > of existing
> > > > > server,
> > > > > > which
> > > > > > > used to be
> > > > > > > > one thread per
> connection
> > classic
> > > model.
> > > > During load
> > > > > > test we
> > > > > > > noticed
> > > > > > > > that NIO server consumes
> > twice
> > > CPU power than
> > > > > the classic
> > > > > > > model when
> > > > > > > > serving the same
> amount of
> > > connections.
> > > > > > > >
> > > > > > > > Here's our
> configurations:
> > > > > > > >
> > > > > > > > Here are some
> statistics:
> > > > > > > >
> > > > > > > > Hardware: Dells 2950
> with
> > 16 GB
> > > RAM, total 8
> > > > > > processors, Intel(R)
> > > > > > > > Xeon(R) CPU E5345 @
> > 2.33GHz , Linux
> > > > kernel 2.6
> > > > > > > >
> > > > > > > > Software: JDK v1.5.6
> > > > > > > >
> > > > > > > > There are 4 instance
> of the
> > > server application
> > > > > > configured on
> > > > > > > one box
> > > > > > > >
> > > > > > > > Grizzly Configs:
> > > > > > > >
> > > > > > > > 100 max worker
> threads (8K
> > buffer
> > > size)
> > > > > > > >
> > > > > > > > 150 max output
> buffer pool
> > size
> > > (4K buffer
> > > > size)
> > > > > > > >
> > > > > > > > use_direct_buffer = true
> > > > > > > >
> > > > > > > > selector timeout = 500
> > > > > > > >
> > > > > > > > Keep alive is true
> (persistent
> > > connections)
> > > > > > > >
> > > > > > > >
> > > > > > > > Tested with : 7200
> client
> > connections
> > > > > > > >
> > > > > > > > Total CPU usage for
> all 4
> > > instances are
> > > > 44.59% (with
> > > > > > NIO engine)
> > > > > > > >
> > > > > > > > CPU usage is 24.76%
> > (without Nio
> > > engine)
> > > > > > > >
> > > > > > > > So far as memory, NIO
> > server uses
> > > slightly
> > > > less
> > > > > memory in
> > > > > > > this case.
> > > > > > > >
> > > > > > > >
> > > > > > > > The memory consumption
> > could be
> > > linear
> > > > because
> > > > > we keep
> > > > > > lots
> > > > > > > of state
> > > > > > > > information for each
> > connection
> > > (user).
> > > > Other than
> > > > > > that, what
> > > > > > > other
> > > > > > > > parameters should I pay
> > attention
> > > to in
> > > > order to
> > > > > scale the
> > > > > > > server to
> > > > > > > > serve 10,000 or more
> > connections
> > > without
> > > > burning
> > > > > up CPU?
> > > > > > > >
> > > > > > > > The old server can serve
> > 12000
> > > connections
> > > > with 4
> > > > > > instance at
> > > > > > > <= 50%
> > > > > > > > CPU usage.
> > > > > > > >
> > > > > > > > Would it be better
> to run 2
> > > instances of
> > > > NIO server
> > > > > > instead
> > > > > > > of 4?
> > > > > > > >
> > > > > > > > Your opinion and
> suggestion is
> > > greatly
> > > > appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > > Fay
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Charlie Hunt
> > > > > > > Java Performance Engineer
> > > > > > > 630.285.7708 x47708
> (Internal)
> > > > > > >
> > > > > > > <
> > http://java.sun.com/docs/performance/
> > > > > > <
> http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>
> > > > > < http://java.sun.com/docs/performance/
> > > <http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>>>
> > > > > > >
> > > < http://java.sun.com/docs/performance/
> > > < http://java.sun.com/docs/performance/>>>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > >
> > > > > > > To unsubscribe, e-mail:
> > > > > >
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>>
> > > > > > > <mailto:
> > > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>>>
> > > > > > > For additional
> commands, e-mail:
> > > > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>> <mailto:
> > > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>>
> > > > > > > <mailto:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>>>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Charlie Hunt
> > > > > > Java Performance Engineer
> > > > > > 630.285.7708 x47708 (Internal)
> > > > > >
> > > > > > <
> http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>
> > > > > > <
> http://java.sun.com/docs/performance/
> > > < http://java.sun.com/docs/performance/>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > >
> > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > >
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>>
> > > > > > For additional commands, e-mail:
> > > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>> >
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > < http://java.sun.com/docs/performance/
> > > < http://java.sun.com/docs/performance/>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net> >> <mailto:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > < http://java.sun.com/docs/performance/>
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > < http://java.sun.com/docs/performance/>
> > >
> > >
> >
> ---------------------------------------------------------------------
> > >
> > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > For additional commands, e-mail:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net> <mailto:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > >
> > >
> > >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > <http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > For additional commands, e-mail:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> >
> >
>
> --
>
> Charlie Hunt
> Java Performance Engineer
> 630.285.7708 x47708 (Internal)
>
> <http://java.sun.com/docs/performance/>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
>
>
--
Charlie Hunt
Java Performance Engineer
630.285.7708 x47708 (Internal)
<http://java.sun.com/docs/performance/>