Salut,
glassfish_at_javadesktop.org wrote:
> Jeanfrancois,
>
> Thanks for the quick response, and the confirmation of my theory.
>
> My test was a bit contrived, but it was designed to easily recreate a much more realistic scenario that we encountered using GF for the PoC I mentioned. In a particular use case an external client synchronously called a particular service (call it A) hosted by the GF instance. In the course of servicing that request, the service implementation for A synchronously called another service B, which was hosted in that same GF instance. Service B synchronously called service C, etc. There wasn't any type of recursion, it was a synchronous invocation "chain" that reached (or attempted to reach) a depth of 5, client -> A -> B -> C -> D-> E -> F, when it broke. (The client invocation of A obviously ties up one other thread.)
>
Thanks for the explanation.
> It was the combination of fairly nested service composition (a natural result of building more granular services out of finer-grained ones) and the fact that the services were all hosted on the same GF instance that caused the problem. This is a normal usage pattern (I agree that recursive service invocation is not realistic!).
>
> When the problem is encountered, all threads on the invocation "chain" are frozen, until an eventual time-out. This impacts system availability, since no other external (or internal) service invocations can be started.
Agree we must have an automatic discovery of such dead lock. But this is
not simple to implement :-) Unless you decrease the times out, I agree
with you this is not the best situation.
>
> Expanding the thread pool size addresses the case of a single client accessing the service, but the problem returns when multiple clients are concurrently accessing the service. A sufficiently large thread pool would make the probability of exhausting the thread pool lower, but there is no way reduce it to zero. This makes mounting a fairly simple DoS attack quite easy. It also affects scalability, since the number of on-going service invocations is limited by the thread pool size, and threads are an inherently expensive resource. So I'm sure you can understand my concern.
Fully. That's why I've left the bug open. For sure we need to improve
the current scenario.
>
> I'm sure Jetty and Tomcat share this same vulnerability, but I was surprised that Grizzly introduces the same problem. It sounds like there is an opportunity to improve performance within GF by detecting the self-hosted service invocation scenario, and avoiding some of the lower-level parts of the protocol stack, as well as avoiding the need for a new thread. Just a suggestion...
I agree. Although the web service team can probably implement such
mechanism (cc-ing Jitu to see what he thinks), something at the Grizzly
code could also be usefull for other scenario where all the threads
blocks on something (like a database).
>
> By the way, the Open ESB normalized message router (NMR) employs an interested pattern that breaks synchronous service invocation MEPs into asynchronously handled messages (with MEP state associated with them). This is a much more scalable approach (the number of threads is constant, not a function of the number of concurrent service invocations), and might be worth a look as a pattern for improving message handling within GF / Grizzly. Just another thought...
:-) If I recall properly, they are using Grizzly ARP (Asynchronous
Request Processing) (old blog [1]), which allow them to free threads
that block/lock like the current scenario. I would also think this would
be a good decision to implement in GlassFish...just need to evangelize
it internally :-)
Thanks!
-- Jeanfrancois
[1]
http://weblogs.java.net/blog/jfarcand/archive/2006/02/grizzly_part_ii.html
>
> Best regards,
> --Ron
> [Message sent by forum member 'rtenhove' (rtenhove)]
>
> http://forums.java.net/jive/thread.jspa?messageID=297853
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>