Sounds like we might be best to work with the concurrent collector.
How much physical memory do you have on your test machine (and your
production machine) ?
And, do you have any additional applications you are wanting to run on
the same machine at the same time ?
IIRC, I think you were using a 700M java heap?
Can you send your entire java command line?
We'll likely configure the JVM with a rather small young generation with
a large old generation with the premise of promoting "medium lived"
object into old generation and letting the concurrent collector clean
these up. The challenge will be sizing old generation such that we
don't over-run it with new garbage too fast that the concurrent
collector can't free it, (I refer to this as "losing the race".).
charlie ...
Fay Zheng wrote:
> Yes. We care more about the raw throughput. Our main goal is to
> achieve higher maximum capacity through this project. Most of our
> traffic is small keepalive messages, which has a timeout of 30
> seconds. However the desirable response time for game specific
> messages is no more than 500 milliseconds.
>
> I looked at the GC log in more detail, the full GC looks better in
> NIO, but we have a much worse minor GC time for YoungGen with NIO
> engine when compare to old classic gameserver:
>
>
> rserv_log_070817_125402:7436.600: [Full GC [PSYoungGen:
> 5488K->0K(61632K)] [PSOldGen: 41294K->35274K(44992K)]
> 46782K->35274K(106624K) [PSPermGen: 16366K->16366K(26752K)], 0.1906040
> secs]
>
> rserv_log_070817_125402:7442.548: [GC [PSYoungGen:
> 45568K->1504K(61504K)] 80842K->36778K(106496K), 0.0078060 secs]
>
> rserv_log_070817_125402:7448.310: [GC [PSYoungGen:
> 47072K->2112K(63168K)] 82346K->38638K(108160K), 0.0838490 secs]
>
> rserv_log_070817_125402:7449.229: [GC [PSYoungGen:
> 49856K->5912K(53696K)] 86382K->44402K(98688K), 0.6270710 secs]
>
> rserv_log_070817_125402:7450.578: [GC [PSYoungGen:
> 53656K->5856K(64192K)] 92146K->50203K(109184K), 1.0315760 secs]
>
> rserv_log_070817_125402:7451.609: [Full GC [PSYoungGen:
> 5856K->0K(64192K)] [PSOldGen: 44347K->38032K(48896K)]
> 50203K->38032K(113088K) [PSPermGen: 16374K->16374K(26240K)], 0.2052210
> secs]
>
>
>
> There are quiet some minor GCs in NIO is taking more than one second.
> The average minor GC time is 0.206815. While the average time of minor
> GC for none-NIO server is 0.0133541.
>
> Full_GC_baseline avg / max duration in seconds: 0.39 / 0.58
>
>
>
> Full_GC_NIO avg / max duration in seconds: 0.28 / 0.41
>
> thanks,
>
> Fay
>
>
>
> On 8/21/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> wrote:
>
> Hi Fay,
>
> Are you most interested in "raw throughput" with this application with
> little or no interest in "pause time / responsiveness" ?
>
> Or, is "pause time / responsiveness" more important than throughput ?
>
> Do you have a threshold at which you want all your GC pause times
> to be
> less than?
>
> Let's start with that ... then we'll work on tuning gc for your
> NIO server.
>
> charlie ...
>
> Fay Zheng wrote:
> > Now we have a GC problem. The NIO server is using a lot more GC
> time:
> >
> > 1) GC data for NIO server :
> >
> >
> > GC collector: Name = 'PS Scavenge', Collections = 737, Total time
> > spent = 2 minutes
> >
> > GC collector: Name = 'PS MarkSweep', Collections = 59, Total time
> > spent = 16.414 seconds
> >
> > Committed memory: 97, 088 kbytes
> >
> >
> >
> > 1) GC data for none - NIO server :
> >
> > GC collector: Name = 'PS Scavenge', Collections = 1075, Total time
> > spent = 14.349 seconds
> >
> > GC collector: Name = 'PS MarkSweep', Collections = 17, Total time
> > spent = 6.511 seconds
> >
> > Committed memory: 102,464 kbytes
> >
> > And we are using the default GC settings from JVM 1.5.12.
> >
> > Any ideas on how to configure the GC for NIO engine will be greatly
> > appreciated?
> >
> > thanks,
> > Fay
> >
> > On 8/20/07, *Fay Zheng* <fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>> wrote:
> >
> > It seems that the fragmented packet behavior goes away after
> I set
> > tcpNoDelay to false explicitly in the socket connection
> > settings for both client and server.
> >
> > thanks,
> > Fay
> >
> >
> > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> wrote:
> >
> > Any chance you can try your configuration on a platform
> other
> > than Linux
> > such as OpenSolaris (x86) or Windows ?
> >
> > Linux has a history of being a little trouble-some when it
> > comes to
> > working well with Java NIO.
> >
> > This is the first I have heard of someone observing the sort
> > of thing
> > you are describing where you read small portions of data
> at a
> > time.
> >
> > Can you isolate your testing system completely
> independent of
> > all other
> > LAN traffic, (i.e. isolate the test system to include
> only the
> > machine
> > your are testing on a load machine) ?
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Disable TcpNoDelay does not seem to help. We still receive
> > tons of
> > > fragmented packets on single read.
> > >
> > > Has anyone had similar experience before?
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>
> > wrote:
> > >
> > > Hi Fay,
> > >
> > > You could try it. It may help.
> > >
> > > If it does indeed improve things for you, let us know.
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Thank you for your help. I also noticed that
> there are
> > some TCP
> > > > specific behaviors. Looks like tcp connection is
> > sending 1 byte
> > > > (partial data) on one read, then the rest of the
> > request content on
> > > > next read. So we have to assembling the request.
> This
> > happens quiet
> > > > often during our load test. Should I enable "
> > tcpNoDelay"? Will it
> > > > help to turn off the behavior?
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/14/07, *charlie hunt* <
> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>> wrote:
> > > >
> > > > There's a couple places in the grizzly
> source code
> > you would
> > > need to
> > > > explicitly modify from that I looked at
> yesterday.
> > > >
> > > > Here's what I identified:
> > > >
> > > > 1.) In Controller.java , doSelect()
> method. Remove
> > this
> > > logic towards
> > > > the end of the method:
> > > > iterator = readyKeys.iterator();
> > > > long currentTime =
> > System.currentTimeMillis ();
> > > > while (iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > key = iterator.next();
> > > > selectionKeyHandler.expire
> > (key,currentTime);
> > > > }
> > > > *Note: In the trunk, Alexy just moved this
> logic to the
> > > > DefaultSelectionKeyHandler.
> > > >
> > > > 2.) If you are using the TCPSelectorHandler, in
> > onReadOps(),
> > > you don't
> > > > need the currentTime to initialized, you can
> just
> > set it to
> > > 0, or
> > > > remove
> > > > the currentTime and replace currentTime with
> 0L in the
> > > > selectionKeyHandler.register(key, currentTime)
> > call. Same thing
> > > > applies
> > > > to onWriteOps(). In onAcceptInterest(), you
> can
> > replace the
> > > call to
> > > > System.currentTimeMillis () with 0L.
> > > >
> > > > 3.) In DefaultSelectionKeyHandler.expire (),
> just
> > make the entire
> > > > method
> > > > a no-op. Just tell it to return as soon as it is
> > entered.
> > > >
> > > > I think that is it. That is all I could find.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Thank you! We've already override the expire
> > method in the
> > > > > DefaultSelectionKeyHandler to do nothing
> for our
> > application:
> > > > >
> > > > > DefaultSelectionKeyHandler keyHandler =
> > > > >
> > > > > *new* DefaultSelectionKeyHandler() {
> *public* *void*
> > > > > expire(SelectionKey key) { *return*;
> > > > >
> > > > > }
> > > > >
> > > > > };
> > > > >
> > > > > However this approach looks quiet expensive in
> > the profiling
> > > > data. For
> > > > > now I have to customize the grizzly source
> code
> > to remove the
> > > > expiring
> > > > > of key part.
> > > > >
> > > > > thanks,
> > > > > Fay
> > > > >
> > > > >
> > > > > On 8/13/07, *charlie hunt* <
> charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>> wrote:
> > > > >
> > > > > Hi Fay,
> > > > >
> > > > > At the moment, there is not a way to
> turn off
> > expiring
> > > of a
> > > > key.
> > > > >
> > > > > But, it is functionality we should
> have and I
> > know of
> > > > another project
> > > > > which will want this functionality
> disabled
> > as well.
> > > > >
> > > > > I will file an enhancement issue to
> add this
> > > functionality.
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > Is there a way to turn off expire of
> key,
> > since it
> > > is not
> > > > > applicable
> > > > > > to persistent connections:
> > > > > >
> > > > > > iterator = readyKeys.iterator ();
> > > > > >
> > > > > > while ( iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > > >
> > > > > > key = iterator.next();
> > > > > >
> > > > > > selectionKeyHandler.expire(key);
> > > > > >
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Fay
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 8/10/07, *Fay Zheng* <
> > fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>
> <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>
> > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>>
> > > > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>> <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>> <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>>
> > > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto: fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>
> > <mailto:fzheng1998_at_gmail.com
> <mailto:fzheng1998_at_gmail.com>>>>>>> wrote:
> > > > > >
> > > > > > Here's the command to run the
> application:
> > > > > >
> > > > > > java -server -verbose:gc
> -XX:+PrintGC
> > > > > > Details -XX:+PrintGCTimeStamps -
> > > > > >
> > Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > > - Dcom.sun.management.jmxremote
> > > > > > -
> Dcom.sun.management.jmxremote.port=9991
> > > > > >
> > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > >
> -Dcom.sun.management.jmxremote.ssl=false
> > > > > > - Duser.region=US -Xmx700m
> > com.pg.serv.room.Main
> > > > > >
> > > > > > The GC is almost 10% for NIO, 5% for
> > none NIO
> > > server.
> > > > > >
> > > > > > thansk,
> > > > > > Fay
> > > > > >
> > > > > > On 8/8/07, *charlie hunt* <
> > charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>
> > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>
> > > > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>
> > > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> > > > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>> <mailto: charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>
> > <mailto:charlie.hunt_at_sun.com
> <mailto:charlie.hunt_at_sun.com>>>>>>>
> > > > > wrote:
> > > > > >
> > > > > > In addition to "RG's"
> suggestion, I
> > would:
> > > > > >
> > > > > > 1.) Set use_direct_buffer =
> > false (Grizzly
> > > will use
> > > > > > HeapByteBuffer instead)
> > > > > > 2.) You might also reduce the
> > number of
> > > instances
> > > > of the
> > > > > > server down to
> > > > > > 2 or even 1. NIO should
> give you
> > much better
> > > > > scalability and
> > > > > > should
> > > > > > have no problems scaling to
> 12,000
> > connections.
> > > > > > 3.) If you can, I would also
> go to the
> > > latest JDK 6_02
> > > > > version.
> > > > > >
> > > > > > We might also need to do some
> > further fine
> > > tuning
> > > > of the
> > > > > JVM too.
> > > > > >
> > > > > > Could you share with us the
> full java
> > > command line
> > > > args you
> > > > > > are using?
> > > > > > It might also be useful to
> see the
> > output from
> > > > > > -XX:+PrintGCDetails when
> > > > > > you are running with NIO.
> > > > > >
> > > > > > hths,
> > > > > >
> > > > > > charlie ...
> > > > > >
> > > > > > Fay Zheng wrote:
> > > > > > > We add Grizzly Nio engine
> to one
> > of existing
> > > > server,
> > > > > which
> > > > > > used to be
> > > > > > > one thread per connection
> classic
> > model.
> > > During load
> > > > > test we
> > > > > > noticed
> > > > > > > that NIO server consumes
> twice
> > CPU power than
> > > > the classic
> > > > > > model when
> > > > > > > serving the same amount of
> > connections.
> > > > > > >
> > > > > > > Here's our configurations:
> > > > > > >
> > > > > > > Here are some statistics:
> > > > > > >
> > > > > > > Hardware: Dells 2950 with
> 16 GB
> > RAM, total 8
> > > > > processors, Intel(R)
> > > > > > > Xeon(R) CPU E5345 @
> 2.33GHz , Linux
> > > kernel 2.6
> > > > > > >
> > > > > > > Software: JDK v1.5.6
> > > > > > >
> > > > > > > There are 4 instance of the
> > server application
> > > > > configured on
> > > > > > one box
> > > > > > >
> > > > > > > Grizzly Configs:
> > > > > > >
> > > > > > > 100 max worker threads (8K
> buffer
> > size)
> > > > > > >
> > > > > > > 150 max output buffer pool
> size
> > (4K buffer
> > > size)
> > > > > > >
> > > > > > > use_direct_buffer = true
> > > > > > >
> > > > > > > selector timeout = 500
> > > > > > >
> > > > > > > Keep alive is true (persistent
> > connections)
> > > > > > >
> > > > > > >
> > > > > > > Tested with : 7200 client
> connections
> > > > > > >
> > > > > > > Total CPU usage for all 4
> > instances are
> > > 44.59% (with
> > > > > NIO engine)
> > > > > > >
> > > > > > > CPU usage is 24.76%
> (without Nio
> > engine)
> > > > > > >
> > > > > > > So far as memory, NIO
> server uses
> > slightly
> > > less
> > > > memory in
> > > > > > this case.
> > > > > > >
> > > > > > >
> > > > > > > The memory consumption
> could be
> > linear
> > > because
> > > > we keep
> > > > > lots
> > > > > > of state
> > > > > > > information for each
> connection
> > (user).
> > > Other than
> > > > > that, what
> > > > > > other
> > > > > > > parameters should I pay
> attention
> > to in
> > > order to
> > > > scale the
> > > > > > server to
> > > > > > > serve 10,000 or more
> connections
> > without
> > > burning
> > > > up CPU?
> > > > > > >
> > > > > > > The old server can serve
> 12000
> > connections
> > > with 4
> > > > > instance at
> > > > > > <= 50%
> > > > > > > CPU usage.
> > > > > > >
> > > > > > > Would it be better to run 2
> > instances of
> > > NIO server
> > > > > instead
> > > > > > of 4?
> > > > > > >
> > > > > > > Your opinion and suggestion is
> > greatly
> > > appreciated.
> > > > > > >
> > > > > > >
> > > > > > > thanks,
> > > > > > > Fay
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Charlie Hunt
> > > > > > Java Performance Engineer
> > > > > > 630.285.7708 x47708 (Internal)
> > > > > >
> > > > > > <
> http://java.sun.com/docs/performance/
> > > > > < http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > > >
> > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> >
> > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>>
> > > > > > For additional commands, e-mail:
> > > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>> <mailto:
> > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > > > <mailto:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > < http://java.sun.com/docs/performance/
> > > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> >
> > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net> >
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>>
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > < http://java.sun.com/docs/performance/
> > <http://java.sun.com/docs/performance/>>
> > > >
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > <mailto:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>> <mailto:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto:dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > < http://java.sun.com/docs/performance/>
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > For additional commands, e-mail:
> > dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>>
> > >
> > >
> >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > <http://java.sun.com/docs/performance/>
> >
> >
> ---------------------------------------------------------------------
> >
> > To unsubscribe, e-mail:
> dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > For additional commands, e-mail:
> dev-help_at_grizzly.dev.java.net <mailto:dev-help_at_grizzly.dev.java.net>
> > <mailto: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>>
> >
> >
> >
>
> --
>
> Charlie Hunt
> Java Performance Engineer
> 630.285.7708 x47708 (Internal)
>
> <http://java.sun.com/docs/performance/
> <http://java.sun.com/docs/performance/>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> <mailto:dev-help_at_grizzly.dev.java.net>
>
>
--
Charlie Hunt
Java Performance Engineer
630.285.7708 x47708 (Internal)
<http://java.sun.com/docs/performance/>