users@grizzly.java.net

Re: Fairness of request processing in Glassfish

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Thu, 22 Nov 2007 10:43:52 -0500

Salut,

Jesper Söderlund wrote:
> Hmm, yes those two articles certainly point the way.
>
> Any idea when a RCM functionality would be a supported option in Glassfish?

It is a supported options in v2, but not enabled by default. The
configuration part needs to be improved (via admin-gui)..I agree :-)


>
> Some comments on the policies:
> - reservation - This means that we have to over-allocate resources which
> are then partitioned into separate "pools", this way to allocating a
> percentage to each app is rather cumbersome to manage (even if there was
> a snazzy GUI instead of the system properties) since the balance will
> need to be reconfigured with each new application deployed.

Indeed. Every request to a newly deployed application will be rejected
(503) if the other deployed applications have reserved 100% of the
thread pool. But you can always leave a small amount of the thread count
available (let's say 10%).

>
> - ceiling - Unfortunatly I did not fully understand the semantics of the
> ceiling policy, but I venture a guess that it's a common pool and that
> an app is maximally allowed a certain percentage allocation from that
> pool, else it will block for a while, this seems quite restrictive but
> is effective at blocking "run-away" applications so it could be useful
> to set the ceiling quite high to prevent DDoS type of attacks (or bugs
> with the similar effect)

Right. The main difference between the two is with reservation, if App A
has 30% allocated threads, App B 50%, and 20% left for other
application, if App A needs more that 30%, it can always use the 20%
unallocated. With ceiling, if 30% of the threads are used, then requests
to App A will not be executed (but instead parked...similar to a
continuation).

>
> I'd also like to see a pool with a weighted policy where each app is
> allocated a weight and the resources allocated according to that weight,

If I understand you correctly, one thread pool but a dispatcher that
pick up threads from the pool using the weight?


> that would be much more robust, in terms of administration at least,
> incidentally this is also one of the algorithm available in Solaris
> resource management, perhaps some lessons can be learned from that?

Agree. The RCM stuff is probably really similar to zones as well.

The
> weight per application should be configurable in the deployment
> description as well as in the admin GUI to override the deployment
> descriptor value. Applications which don't have a value would be
> assigned a default value.
>

Agree. I've filled this but never has a chance to fix it:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=3058

> Though I understand the motivation for having a default policy which
> optimizes on performance as that is quite visible in benchmarks, etc,
> but I think it would better if an policy which would prioritize fairness
> between applications would be just as easily configured or maybe even
> the default and the performance policy the exception which is configured
> for benchmark tests.

Agree. You should reply to the v3 discussion in GlassFish asking to have
rcm enabled by default and better supported. I will speak on my side as
well :-)

A+

-- Jeanfrancois

>
> cheers,
> /Jesper
>
>
>
> 2007/11/21, Jeanfrancois Arcand <Jeanfrancois.Arcand_at_sun.com
> <mailto:Jeanfrancois.Arcand_at_sun.com>>:
>
> Salut,
>
> Jesper Söderlund wrote:
> > I wonder if somebody could enlighten me with regards to the
> fairness of
> > processing requests in Glassfish / Grizzly under load.
>
> By default, same fairness for every requests are applied, meaning the
> first requests that comes in is always the first executed (something
> Tomcat doesn't do). This is quite important and one of the reason why we
> have good score with specJ2004: all requests have the same chance to be
> executed. This is the SEDA part of Grizzly that allow use to do that :-)
>
> >
> > Is there some some guarantee that one heavily loaded application will
> > not hog all the resources from another using the same grizzly
> instance?
>
> By default, if an application takes more CPU/resources than the other,
> Grizzly doesn't have a mechanism to isolate the application by default.
> I said by default because Grizzly have a extension (with both 1.0 and
> 1.5) that allow resource consumptions management [1]. This is something
> you should take alook.
>
> >
> > Is the only way to "partion" the resources to create separate HTTP
> > listeners?
>
> No, see [2] for a good example.
>
> >
> > Will these threads configured for the HTTP-listener run through the
> > servlet container and down into the EJB-container?
>
> Yes.
>
> >
> > It would be great to be able to specify a weight for an
> application in
> > the deployment descriptor and have the request processing scheduled
> > accordingly.
>
> Look at [1][2] :-) This is exactly what you need. 4homemedia.com
> <http://4homemedia.com>
> combined [2] with Grizzly Comet for their application and they told me
> their performance has improved by isolating applications.
>
> Thanks!
>
> -- Jeanfrancois
>
> [1]
> http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html
> <http://weblogs.java.net/blog/jfarcand/archive/2006/04/resource_consum.html>
> [2]
> http://weblogs.java.net/blog/jfarcand/archive/2007/06/improving_ajax_1.html
>
> >
> > /Jesper
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> <mailto:users-unsubscribe_at_grizzly.dev.java.net>
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
> <mailto:users-help_at_grizzly.dev.java.net>
>
>