users@grizzly.java.net

Re: grizzly-comet-webserver gets stuck sometime while processing request

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 17 Jun 2009 16:15:23 -0400

Salut,

I've filled:

https://grizzly.dev.java.net/issues/show_bug.cgi?id=672

for the tracking the performance regression when readThread > 0

A+

-- Jeanfrancois

Jeanfrancois Arcand wrote:
> Salut,
>
> Great analysis as usual!
>
> if we set the processorTask to null for every request, we will regress
> in term of performance as we will need to re-create/sync a new instance.
> If DefaultProtocolFilter, we have:
>
>> 134 if (streamAlgorithm.parse(byteBuffer)){
>> 135 136 if (processorTask == null){
>> 137 processorTask = selectorThread.getProcessorTask();
>> 138 workerThread.setProcessorTask(processorTask);
>> 139 }
>
> For normal processing, we set that value once and re-use it. For ARP and
> Comet, we are using the c.s.g.arp.AsyncProtocolFilter, and we are doing:
>
>> 162 if (streamAlgorithm.parse(byteBuffer)){
>> 163 ProcessorTask processor =
>> selectorThread.getProcessorTask();
>> 164 configureProcessorTask(processor, ctx,
>> 165 streamAlgorithm.getHandler(), inputStream);
>
> we never get the ProcessorTask from the HttpWorkerThread. So I think
> your patch fix the issue but it just a bug when readerThread > 1. I
> suspect (I will take a look as I might be wrong) the AsyncProtocolFilter
> is not added to the ProtocolChain because the ARP is not enabled, hence
> we ends up with the DefaultProtocolFilter instead of AsyncProtocolFilter.
>
> A+
>
> -- Jeanfrancois
>
>
>
> Bongjae Chang wrote:
>> Hi Abey,
>>
>> I attached the proposed patch experimentally.
>>
>> I added HttpWorkerThread#reset() for cleaning up used thread.
>>
>> When I applied it, the problem never happened.
>>
>> Could you apply this patch and test it again?
>>
>> --
>> Bongjae Chang
>>
>>
>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>> To: <users_at_grizzly.dev.java.net>
>> Sent: Wednesday, June 17, 2009 8:45 PM
>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>> processing request
>>
>>
>>> Hi,
>>>
>>> After all, I can reproduce it and I attach HttpWatch's snapshots.
>>>
>>> I found the machine which was reproduced. :)
>>>
>>> At first, I suspect DefaultThreadPool and FixedThreadPool. So I tried
>>> to change the thread pool into JDKThreadPool.
>>>
>>> But this problem kept still.
>>>
>>> As you see the attched snapshot, whenever this problem is reproduced,
>>> the http response is pushed.
>>>
>>> Here is example.
>>>
>>> In stuck.jpg,
>>> ---
>>> req: body-background.png
>>> res: none
>>>
>>> req: main-background.png
>>> res: body-background.png
>>>
>>> req: header-background.png
>>> res: main-background.png
>>> ---
>>>
>>> Maybe when request processing is very quick, it seems that concurrent
>>> problem exists about responsing the http header and body. But I am
>>> not sure.
>>>
>>> I am also curious to know how this phenomenon can happen and I will
>>> try to analyze this as well curiously.
>>>
>>> But if anybody knows the reason, please advice me.
>>>
>>> Thanks.
>>>
>>> --
>>> Bongjae Chang
>>>
>>>
>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>> To: <users_at_grizzly.dev.java.net>
>>> Sent: Wednesday, June 17, 2009 6:05 PM
>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>> processing request
>>>
>>>
>>>> Hi Abey,
>>>>
>>>> Thank you for testing it again.
>>>>
>>>> So now, maybe two issue exists.
>>>>
>>>> 1. processing request get stuck forever
>>>> 2. Slow request issue
>>>>
>>>> Abey wrote:
>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>> delay in
>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>> I understood. Thank you for useful information.
>>>>
>>>> And I am trying to reproduce your issue continually.
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Bongjae Chang
>>>>
>>>>
>>>> ----- Original Message ----- From: "Abey Tom" <appu_abey_tom_at_yahoo.com>
>>>> To: <users_at_grizzly.dev.java.net>
>>>> Sent: Wednesday, June 17, 2009 5:48 PM
>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>> processing request
>>>>
>>>>
>>>>> I'm facing both stuck forever request(never released) and slow
>>>>> request issues
>>>>> even when I use the default thread pool configuration (min/max - 5/5).
>>>>> I tested with the scenarios with 1.
>>>>> -Dcom.sun.grizzly.maxSelectorReadThread = 0 : The request
>>>>> processing is
>>>>> very quick but requests are getting stuck more often.
>>>>> 2. -Dcom.sun.grizzly.maxSelectorReadThread = 1 or 2 : The request
>>>>> processing is slower but some requests seems to get stuck but they are
>>>>> released. But STILL some requests are stuck forever.
>>>>>
>>>>> The chances of getting STUCK are MORE in Powerful server class
>>>>> machines(4
>>>>> core , 64bit, Linux) than in my laptop(Single core,32 bit , Windows ).
>>>>>
>>>>> Also abstract from first mail:
>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>> delay in
>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>
>>>>> method DefaultThreadPool.execute(Runnable){ .... if (running){
>>>>> try { Thread.sleep(2);
>>>>> //or Thread.sleep(5);
>>>>> } catch {....} workQueue.offer(task);
>>>>> queueSize.incrementAndGet(); } }
>>>>>
>>>>> Thanks,
>>>>> Abey Tom
>>>>>
>>>>>
>>>>> BongJaeChang wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> When I tested it with various situations, I could meet some strange
>>>>>>> behavior.
>>>>>>>
>>>>>>> When I used the read thread count which was not 0, processing
>>>>>>> request was
>>>>>>> very slow. For a while, it seemed getting stuck, but it always was
>>>>>>> released.
>>>>>>>
>>>>>>> At first, I thought that the behavior was normal. But when I set
>>>>>>> the read
>>>>>>> thread count to be 0, I could know that processing request is
>>>>>>> very quick.
>>>>>>>
>>>>>>> So, 1. Could you please test it again with
>>>>>>> "-Dcom.sun.grizzly.maxSelectorReadThread=0" ?
>>>>>>>
>>>>>>> 2. And is your problem that processing request is slow or the
>>>>>>> page gets
>>>>>>> stuck and it is never released?
>>>>>>>
>>>>>>> It seems that it is not important the number of the read thread
>>>>>>> count if
>>>>>>> it is not 0 because I can always meet the symptom with setting
>>>>>>> the read
>>>>>>> thread count to be even 1.
>>>>>>>
>>>>>>> And the number of worker threads is not also important because it
>>>>>>> can see
>>>>>>> that the number has changed adequately. But it is not sure.
>>>>>>>
>>>>>>> So I am also debugging it in order to know why the page is slow
>>>>>>> when I
>>>>>>> set the read thread count to be even 1.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Bongjae Chang
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/grizzly-comet-webserver-gets-stuck-sometime-while-processing-request-tp24033280p24069466.html
>>>>>
>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>> --------------------------------------------------------------------------------
>>
>>
>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>> ------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>