Now we have a GC problem. The NIO server is using a lot more GC time:
1) GC data for NIO server :
GC collector: Name = 'PS Scavenge', Collections = 737, Total time spent = 2
minutes
GC collector: Name = 'PS MarkSweep', Collections = 59, Total time spent =
16.414 seconds
Committed memory: 97, 088 kbytes
1) GC data for none - NIO server :
GC collector: Name = 'PS Scavenge', Collections = 1075, Total time spent =
14.349 seconds
GC collector: Name = 'PS MarkSweep', Collections = 17, Total time spent =
6.511 seconds
Committed memory: 102,464 kbytes
And we are using the default GC settings from JVM 1.5.12.
Any ideas on how to configure the GC for NIO engine will be greatly
appreciated?
thanks,
Fay
On 8/20/07, Fay Zheng <fzheng1998_at_gmail.com> wrote:
>
> It seems that the fragmented packet behavior goes away after I set
> tcpNoDelay to false explicitly in the socket connection settings for both
> client and server.
>
> thanks,
> Fay
>
>
> On 8/17/07, charlie hunt <charlie.hunt_at_sun.com> wrote:
> >
> > Any chance you can try your configuration on a platform other than Linux
> > such as OpenSolaris (x86) or Windows ?
> >
> > Linux has a history of being a little trouble-some when it comes to
> > working well with Java NIO.
> >
> > This is the first I have heard of someone observing the sort of thing
> > you are describing where you read small portions of data at a time.
> >
> > Can you isolate your testing system completely independent of all other
> > LAN traffic, (i.e. isolate the test system to include only the machine
> > your are testing on a load machine) ?
> >
> > charlie ...
> >
> > Fay Zheng wrote:
> > > Disable TcpNoDelay does not seem to help. We still receive tons of
> > > fragmented packets on single read.
> > >
> > > Has anyone had similar experience before?
> > >
> > > thanks,
> > > Fay
> > >
> > >
> > > On 8/17/07, *charlie hunt* <charlie.hunt_at_sun.com
> > > <mailto:charlie.hunt_at_sun.com>> wrote:
> > >
> > > Hi Fay,
> > >
> > > You could try it. It may help.
> > >
> > > If it does indeed improve things for you, let us know.
> > >
> > > charlie ...
> > >
> > > Fay Zheng wrote:
> > > > Thank you for your help. I also noticed that there are some TCP
> > > > specific behaviors. Looks like tcp connection is sending 1 byte
> > > > (partial data) on one read, then the rest of the request content
> > on
> > > > next read. So we have to assembling the request. This happens
> > quiet
> > > > often during our load test. Should I enable " tcpNoDelay"? Will
> > it
> > > > help to turn off the behavior?
> > > >
> > > > thanks,
> > > > Fay
> > > >
> > > >
> > > > On 8/14/07, *charlie hunt* <charlie.hunt_at_sun.com
> > > <mailto:charlie.hunt_at_sun.com>
> > > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>
> > wrote:
> > > >
> > > > There's a couple places in the grizzly source code you would
> >
> > > need to
> > > > explicitly modify from that I looked at yesterday.
> > > >
> > > > Here's what I identified:
> > > >
> > > > 1.) In Controller.java , doSelect() method. Remove this
> > > logic towards
> > > > the end of the method:
> > > > iterator = readyKeys.iterator();
> > > > long currentTime = System.currentTimeMillis ();
> > > > while (iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > key = iterator.next();
> > > > selectionKeyHandler.expire (key,currentTime);
> > > > }
> > > > *Note: In the trunk, Alexy just moved this logic to the
> > > > DefaultSelectionKeyHandler.
> > > >
> > > > 2.) If you are using the TCPSelectorHandler, in onReadOps(),
> >
> > > you don't
> > > > need the currentTime to initialized, you can just set it to
> > > 0, or
> > > > remove
> > > > the currentTime and replace currentTime with 0L in the
> > > > selectionKeyHandler.register(key, currentTime) call. Same
> > thing
> > > > applies
> > > > to onWriteOps(). In onAcceptInterest(), you can replace the
> > > call to
> > > > System.currentTimeMillis () with 0L.
> > > >
> > > > 3.) In DefaultSelectionKeyHandler.expire(), just make the
> > entire
> > > > method
> > > > a no-op. Just tell it to return as soon as it is entered.
> > > >
> > > > I think that is it. That is all I could find.
> > > >
> > > > charlie ...
> > > >
> > > > Fay Zheng wrote:
> > > > > Thank you! We've already override the expire method in the
> >
> > > > > DefaultSelectionKeyHandler to do nothing for our
> > application:
> > > > >
> > > > > DefaultSelectionKeyHandler keyHandler =
> > > > >
> > > > > *new* DefaultSelectionKeyHandler() { *public* *void*
> > > > > expire(SelectionKey key) { *return*;
> > > > >
> > > > > }
> > > > >
> > > > > };
> > > > >
> > > > > However this approach looks quiet expensive in the
> > profiling
> > > > data. For
> > > > > now I have to customize the grizzly source code to remove
> > the
> > > > expiring
> > > > > of key part.
> > > > >
> > > > > thanks,
> > > > > Fay
> > > > >
> > > > >
> > > > > On 8/13/07, *charlie hunt* < charlie.hunt_at_sun.com
> > > <mailto: charlie.hunt_at_sun.com>
> > > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com >>
> > > > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>>>
> > wrote:
> > > > >
> > > > > Hi Fay,
> > > > >
> > > > > At the moment, there is not a way to turn off expiring
> >
> > > of a
> > > > key.
> > > > >
> > > > > But, it is functionality we should have and I know of
> > > > another project
> > > > > which will want this functionality disabled as well.
> > > > >
> > > > > I will file an enhancement issue to add this
> > > functionality.
> > > > >
> > > > > charlie ...
> > > > >
> > > > > Fay Zheng wrote:
> > > > > > Is there a way to turn off expire of key, since it
> > > is not
> > > > > applicable
> > > > > > to persistent connections:
> > > > > >
> > > > > > iterator = readyKeys.iterator();
> > > > > >
> > > > > > while (iterator.hasNext() &&
> > > selectorHandler.isOpen()) {
> > > > > >
> > > > > > key = iterator.next();
> > > > > >
> > > > > > selectionKeyHandler.expire(key);
> > > > > >
> > > > > >
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Fay
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 8/10/07, *Fay Zheng* < fzheng1998_at_gmail.com
> > > <mailto:fzheng1998_at_gmail.com>
> > > > <mailto:fzheng1998_at_gmail.com <mailto: fzheng1998_at_gmail.com>>
> > > > > <mailto: fzheng1998_at_gmail.com
> > > <mailto: fzheng1998_at_gmail.com> <mailto:fzheng1998_at_gmail.com
> > > <mailto:fzheng1998_at_gmail.com>>>
> > > > > > <mailto: fzheng1998_at_gmail.com
> > > <mailto:fzheng1998_at_gmail.com>
> > > > <mailto: fzheng1998_at_gmail.com <mailto:fzheng1998_at_gmail.com>>
> > > <mailto:fzheng1998_at_gmail.com <mailto: fzheng1998_at_gmail.com>
> > > > <mailto: fzheng1998_at_gmail.com
> > > <mailto:fzheng1998_at_gmail.com>>>>> wrote:
> > > > > >
> > > > > > Here's the command to run the application:
> > > > > >
> > > > > > java -server -verbose:gc -XX:+PrintGC
> > > > > > Details -XX:+PrintGCTimeStamps -
> > > > > > Dcom.pg.env=/props/env/dev/pg/trx.properties
> > > > > > -Dcom.sun.management.jmxremote
> > > > > > - Dcom.sun.management.jmxremote.port=9991
> > > > > > -
> > Dcom.sun.management.jmxremote.authenticate=false
> > > > > > -Dcom.sun.management.jmxremote.ssl=false
> > > > > > - Duser.region=US -Xmx700m com.pg.serv.room.Main
> > > > > >
> > > > > > The GC is almost 10% for NIO, 5% for none NIO
> > > server.
> > > > > >
> > > > > > thansk,
> > > > > > Fay
> > > > > >
> > > > > > On 8/8/07, *charlie hunt* < charlie.hunt_at_sun.com
> > > <mailto:charlie.hunt_at_sun.com>
> > > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > > > <mailto:charlie.hunt_at_sun.com
> > > <mailto: charlie.hunt_at_sun.com> <mailto:charlie.hunt_at_sun.com
> > > <mailto:charlie.hunt_at_sun.com >>>
> > > > > > <mailto: charlie.hunt_at_sun.com
> > > <mailto:charlie.hunt_at_sun.com>
> > > > <mailto:charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>>
> > > <mailto: charlie.hunt_at_sun.com <mailto:charlie.hunt_at_sun.com>
> > > > <mailto:charlie.hunt_at_sun.com <mailto: charlie.hunt_at_sun.com
> > >>>>>
> > > > > wrote:
> > > > > >
> > > > > > In addition to "RG's" suggestion, I would:
> > > > > >
> > > > > > 1.) Set use_direct_buffer = false (Grizzly
> > > will use
> > > > > > HeapByteBuffer instead)
> > > > > > 2.) You might also reduce the number of
> > > instances
> > > > of the
> > > > > > server down to
> > > > > > 2 or even 1. NIO should give you much
> > better
> > > > > scalability and
> > > > > > should
> > > > > > have no problems scaling to 12,000
> > connections.
> > > > > > 3.) If you can, I would also go to the
> > > latest JDK 6_02
> > > > > version.
> > > > > >
> > > > > > We might also need to do some further fine
> > > tuning
> > > > of the
> > > > > JVM too.
> > > > > >
> > > > > > Could you share with us the full java
> > > command line
> > > > args you
> > > > > > are using?
> > > > > > It might also be useful to see the output
> > from
> > > > > > -XX:+PrintGCDetails when
> > > > > > you are running with NIO.
> > > > > >
> > > > > > hths,
> > > > > >
> > > > > > charlie ...
> > > > > >
> > > > > > Fay Zheng wrote:
> > > > > > > We add Grizzly Nio engine to one of
> > existing
> > > > server,
> > > > > which
> > > > > > used to be
> > > > > > > one thread per connection classic model.
> > > During load
> > > > > test we
> > > > > > noticed
> > > > > > > that NIO server consumes twice CPU power
> > than
> > > > the classic
> > > > > > model when
> > > > > > > serving the same amount of connections.
> > > > > > >
> > > > > > > Here's our configurations:
> > > > > > >
> > > > > > > Here are some statistics:
> > > > > > >
> > > > > > > Hardware: Dells 2950 with 16 GB RAM, total
> > 8
> > > > > processors, Intel(R)
> > > > > > > Xeon(R) CPU E5345 @ 2.33GHz , Linux
> > > kernel 2.6
> > > > > > >
> > > > > > > Software: JDK v1.5.6
> > > > > > >
> > > > > > > There are 4 instance of the server
> > application
> > > > > configured on
> > > > > > one box
> > > > > > >
> > > > > > > Grizzly Configs:
> > > > > > >
> > > > > > > 100 max worker threads (8K buffer size)
> > > > > > >
> > > > > > > 150 max output buffer pool size (4K buffer
> > > size)
> > > > > > >
> > > > > > > use_direct_buffer = true
> > > > > > >
> > > > > > > selector timeout = 500
> > > > > > >
> > > > > > > Keep alive is true (persistent
> > connections)
> > > > > > >
> > > > > > >
> > > > > > > Tested with : 7200 client connections
> > > > > > >
> > > > > > > Total CPU usage for all 4 instances are
> > > 44.59% (with
> > > > > NIO engine)
> > > > > > >
> > > > > > > CPU usage is 24.76% (without Nio engine)
> > > > > > >
> > > > > > > So far as memory, NIO server uses slightly
> > > less
> > > > memory in
> > > > > > this case.
> > > > > > >
> > > > > > >
> > > > > > > The memory consumption could be linear
> > > because
> > > > we keep
> > > > > lots
> > > > > > of state
> > > > > > > information for each connection (user).
> > > Other than
> > > > > that, what
> > > > > > other
> > > > > > > parameters should I pay attention to in
> > > order to
> > > > scale the
> > > > > > server to
> > > > > > > serve 10,000 or more connections without
> > > burning
> > > > up CPU?
> > > > > > >
> > > > > > > The old server can serve 12000 connections
> >
> > > with 4
> > > > > instance at
> > > > > > <= 50%
> > > > > > > CPU usage.
> > > > > > >
> > > > > > > Would it be better to run 2 instances of
> > > NIO server
> > > > > instead
> > > > > > of 4?
> > > > > > >
> > > > > > > Your opinion and suggestion is greatly
> > > appreciated.
> > > > > > >
> > > > > > >
> > > > > > > thanks,
> > > > > > > Fay
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Charlie Hunt
> > > > > > Java Performance Engineer
> > > > > > 630.285.7708 x47708 (Internal)
> > > > > >
> > > > > > < http://java.sun.com/docs/performance/
> > > > > <http://java.sun.com/docs/performance/
> > > > < http://java.sun.com/docs/performance/>>
> > > > > > <http://java.sun.com/docs/performance/ >>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > > > <mailto:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>>>
> > > > > > For additional commands, e-mail:
> > > > > dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>> <mailto:
> > > > dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>>>
> > > > > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Charlie Hunt
> > > > > Java Performance Engineer
> > > > > 630.285.7708 x47708 (Internal)
> > > > >
> > > > > <http://java.sun.com/docs/performance/
> > > > > < http://java.sun.com/docs/performance/>>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net>>>
> > > > > For additional commands, e-mail:
> > > > dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > > <mailto:dev-help_at_grizzly.dev.java.net
> > > <mailto: dev-help_at_grizzly.dev.java.net>>
> > > > > <mailto:dev-help_at_grizzly.dev.java.net
> > > <mailto: dev-help_at_grizzly.dev.java.net>
> > > > <mailto:dev-help_at_grizzly.dev.java.net
> > > <mailto: dev-help_at_grizzly.dev.java.net>>>
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Charlie Hunt
> > > > Java Performance Engineer
> > > > 630.285.7708 x47708 (Internal)
> > > >
> > > > < http://java.sun.com/docs/performance/ >
> > > >
> > > >
> > > >
> > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > > <mailto: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>>
> > > > For additional commands, e-mail:
> > > dev-help_at_grizzly.dev.java.net <mailto:
> > dev-help_at_grizzly.dev.java.net>
> > > > <mailto: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>>
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Charlie Hunt
> > > Java Performance Engineer
> > > 630.285.7708 x47708 (Internal)
> > >
> > > <http://java.sun.com/docs/performance/>
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> > > <mailto:dev-unsubscribe_at_grizzly.dev.java.net>
> > > For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> > > <mailto:dev-help_at_grizzly.dev.java.net>
> > >
> > >
> >
> >
> > --
> >
> > Charlie Hunt
> > Java Performance Engineer
> > 630.285.7708 x47708 (Internal)
> >
> > <http://java.sun.com/docs/performance/>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> > For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
> >
> >
>