users@grizzly.java.net

Re: Grizzly production ready? ConcurrentHandler & scenarios

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Fri, 13 Mar 2009 17:27:24 -0400

Salut,

felixx wrote:
> Hi all,
> I'm trying to decide on a Comet server side solution. My first preference
> would be to go GF V3 and Grizzly. However based on the existing issues list
> (475, 478, 479) I do have some concerns regarding the stability/maturity of

475: bad test (just recently added, OS dependent failure)
478: Javascript problem (and i'm quite bad with javascript)
479: Maybe an issue with cometd in GF v3.

So nothing related to Comet (server side) :-). We lacked internal
testing for awhile (situation fixed now :-)). Grizzly Comet is stable
but true some minor release 1.9.x might introduce minor regressions).

If your looking at an immature but nice Comet Framework, look at
Atmosphere.dev.java.net (shameless plug as I'm releasing 0.1 today :-))


> the Comet implementation. Here are some questions please:
> 1.Do you guys know/used the Grizzly Comet in a real production env. I know,
> small examples with a few clients would work but how about having 5000
> clients and 2000 messages/s? Has anyone tested/benchmarked such a scenario.

That will be the topic of my javaone session this year :-) ...yes we
have done some test with that but it really depends on the application
itself, long poll vs http streaming, etc. We have two automated
benchmark inside the workspace:

https://grizzly.dev.java.net/nonav/xref-test/com/sun/grizzly/comet/CometUnitTest.html

which can always be looked at. Of course they aren't real benchmark with
a real application.


> It would be unacceptable for us to have the app hanging or being unreliable
> (see the above issues) I've seen some references about GF performance but
> I'm referring strictly to the Comet side. Not easy to say it but, and I may
> be wrong, it looks like Jetty has been used in such scenarios, worked fine
> and seems more mature in this area.

Difficult to convince you here as Jetty was the first one on the market,
and at the time GlassFish was just moving from its "zero" to its "hero"
role :-). But the way Jetty handle suspend/resume cannot scale as every
time you suspend, it throws an exception and your Servlet ends up being
invoked twice. I like Jetty, but the way suspend/resume has been
implemented cannot scale IMO and the logic needed inside a Servlet
(double invokation) make the programming model complicated.

Not only that, but with Jetty you don't have any support detecting when
clients close connection (which can cause memory leak with some
application because of wasted resources), something grizzly supports.

Also the way information/message are pushed in Jetty has to be
implemented by the application itself, where in Grizzly you can easily
filter/throttle/push/cluster using the CometContext.notify() and
NotificationHandler. This is less work to do for an application's developer.


> 2.Maybe I'm missing something here but is there a way to avoid recreating a
> CometHandler every time the same client reconnects for a long pool?

You don't have to recreate. Just re-add it to the list of suspended
response (CometContext.addCometHandler())

  I would
> need a ConcurrentHandler type (with the internal event queue) and seems
> expensive to have for each poll cycle thousands of handlers being created
> and then dropped when an event has ocurred. Why not just being able to
> replace the embedded Response and keep the existing handler as long as the
> client is present?

CometHandler.attach() is for that. But you still need to invoke
resumeCometHandler followed by addCometHandler()


> 3.Here's a scenario I've seen using a different 'long polling' solution. Due
> to a network condition the server is not able to push the events and these
> are accumulating in the handler's queue. (not always the server is able to
> see right away that the client is not able to receive the events). The
> client has a mechanism to detect if ping events are received and if not will
> try again to connect. Now you will end having 2 handler for the same client,
> the old one with some events in the queue, also blocking a thread, and the
> fresh one. How can we handle this type of situations in? In a different
> system I used just to detect the situation, replace the Response and start
> pushing the content of the queue.

why do you think creating CometHandler are expensive? Event if they are
(based on your application), you can alway use the CometHandler.attach()
to recycle them. Did I misunderstood you?


> 4.Even with a ConcurrentHandler you may end having a lot of threads blocked
> because of slow clients. It looks like if you expect 5000 clients it's safer
> to use let's say 400 threads to execute the handler. Any comments on this?

It's up to your application to either:

+ enable async http write in Grizzly (so no blocking thread) insid ethe
application (grizzly internal will handle the "blocking" for you instead).
+ define a "strategy" inside NotificationHandler to detect such slow
client and either discard them or park them inside a "queue" for later
write.

Note that the same issue will arise with Jetty :-)


> A lot of questions, I know. Again GF & Grizzly looks very promising, great
> work so far, I'm not just sure, and I need your help to convince myself, it
> is ready for a real life application.

You are welcome. We have multiples implementation inside and outside Sun
that build on top of Comet. One of them is our internal Sun IM
product...so there is a possibility of ~30 000 users at the same time
using it :-)

Feel free to continue the discussion if I was unclear of if I missed the
point.

A+

-- Jeanfrancois




> Thanks!