users@grizzly.java.net

Re: Stuck Threads, Custom Threadpool

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Tue, 13 Nov 2012 21:59:48 +0100

On 11/13/2012 07:39 PM, Arens, Marc wrote:
>> Oleksiy Stashok <oleksiy.stashok_at_oracle.com> hat am 13. November 2012
>> um 18:35 geschrieben:
>>
>> Hi Marc,
>>
>> On 11/13/2012 05:39 PM, Arens, Marc wrote:
>>> Hello,
>>>
>>> we are currently dealing with incoming ServletRequests getting stuck
>>> in calls to thirdparty libs and filling the FixedThreadPool Grizzly
>>> uses. I have already implement a RequestWatcher that keeps track of
>>> requests/threads that are currently being handled. This
>>> Requestwatcher reports the stuck requests and tries to interrupt
>>> them via Thread.interrupt() and sends an error to the client to
>>> ensure operation of the server. "Of course" those threads don't
>>> react to interrupts and remain in TIMED_WAITING state.
>> That is strange. There is no way to may thread immune wrt
>> Thread.interrupt(), probably something is wrong in 3rd party lib...
>> May be they fall into never-ending loop, or swallow
>> IO)InterruptedException...
>>
>>
>>> Does Grizzly offer a more general approach to remove those threads
>>> from the threadpool? Is it possible to use our own existing
>>> implementation of ThreadPool/ExecutorService to achieve this?
>> Even though we suggest to use Grizzly thread pool implementation,
>> it's possible to set custom executor service to Grizzly
>> Transport#setWorkerThreadPool(ExecutorService).
>> But as I said I doubt it's a problem in Grizzly thread pool
>> implementation.
> Sorry if i wasn't precise enough. The problem isn't located within
> Grizzly but clearly in the 3rd party lib. The InterruptedExceptions
> simply get ignored. I'm just looking for a _central_ point to handle
> such ugly behaviour and make our server more robust against failure as
> we will, sooner or later, face this kind of situations again. I hoped
> Grizzly offered a way to remove these kind of threads from its
> threadpool and stop processing them so it can accept new requests.
> Preventing the threadpool from getting filled with stuck threads ist
> just fighting the symptoms, the requestwatcher will inform us about
> the problem so we can fix the cause.
Unfortunately Grizzly can do nothing about such a threads, even if we
remove it from a thread-pool - this thread will still be active and
shown in the jstack report. Instead of removing such a threads you can
just raise the max-threads-number of Grizzly thread-pool, so it can grow
up to Integer.MAX_VALUE or until it hits some OS/Java limit.

For sure you can try to call thread.stop() to kill the tread, which is
very unsafe.

I can just recommend to file a bug against the component, which is so
unfriendly to (IO)InterruptedException :)

Thanks.

WBR,
Alexey.

> Best regards,
> Marc