users@grizzly.java.net

Re: grizzly-comet-webserver gets stuck sometime while processing request

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 17 Jun 2009 20:31:29 -0400

Salut,

I think commit r3343 fixed the issue (at least I wasn't able to
reproduce it). Can you try and let me know!

Thanks both of you for all the help! That was a nasty one!

-- Jeanfrancois

Jeanfrancois Arcand wrote:
> Salut,
>
> I've filled:
>
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=672
>
> for the tracking the performance regression when readThread > 0
>
> A+
>
> -- Jeanfrancois
>
> Jeanfrancois Arcand wrote:
>> Salut,
>>
>> Great analysis as usual!
>>
>> if we set the processorTask to null for every request, we will regress
>> in term of performance as we will need to re-create/sync a new
>> instance. If DefaultProtocolFilter, we have:
>>
>>> 134 if (streamAlgorithm.parse(byteBuffer)){
>>> 135 136 if (processorTask == null){
>>> 137 processorTask = selectorThread.getProcessorTask();
>>> 138 workerThread.setProcessorTask(processorTask);
>>> 139 }
>>
>> For normal processing, we set that value once and re-use it. For ARP
>> and Comet, we are using the c.s.g.arp.AsyncProtocolFilter, and we are
>> doing:
>>
>>> 162 if (streamAlgorithm.parse(byteBuffer)){
>>> 163 ProcessorTask processor =
>>> selectorThread.getProcessorTask();
>>> 164 configureProcessorTask(processor, ctx,
>>> 165 streamAlgorithm.getHandler(), inputStream);
>>
>> we never get the ProcessorTask from the HttpWorkerThread. So I think
>> your patch fix the issue but it just a bug when readerThread > 1. I
>> suspect (I will take a look as I might be wrong) the
>> AsyncProtocolFilter is not added to the ProtocolChain because the ARP
>> is not enabled, hence we ends up with the DefaultProtocolFilter
>> instead of AsyncProtocolFilter.
>>
>> A+
>>
>> -- Jeanfrancois
>>
>>
>>
>> Bongjae Chang wrote:
>>> Hi Abey,
>>>
>>> I attached the proposed patch experimentally.
>>>
>>> I added HttpWorkerThread#reset() for cleaning up used thread.
>>>
>>> When I applied it, the problem never happened.
>>>
>>> Could you apply this patch and test it again?
>>>
>>> --
>>> Bongjae Chang
>>>
>>>
>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>> To: <users_at_grizzly.dev.java.net>
>>> Sent: Wednesday, June 17, 2009 8:45 PM
>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>> processing request
>>>
>>>
>>>> Hi,
>>>>
>>>> After all, I can reproduce it and I attach HttpWatch's snapshots.
>>>>
>>>> I found the machine which was reproduced. :)
>>>>
>>>> At first, I suspect DefaultThreadPool and FixedThreadPool. So I
>>>> tried to change the thread pool into JDKThreadPool.
>>>>
>>>> But this problem kept still.
>>>>
>>>> As you see the attched snapshot, whenever this problem is
>>>> reproduced, the http response is pushed.
>>>>
>>>> Here is example.
>>>>
>>>> In stuck.jpg,
>>>> ---
>>>> req: body-background.png
>>>> res: none
>>>>
>>>> req: main-background.png
>>>> res: body-background.png
>>>>
>>>> req: header-background.png
>>>> res: main-background.png
>>>> ---
>>>>
>>>> Maybe when request processing is very quick, it seems that
>>>> concurrent problem exists about responsing the http header and body.
>>>> But I am not sure.
>>>>
>>>> I am also curious to know how this phenomenon can happen and I will
>>>> try to analyze this as well curiously.
>>>>
>>>> But if anybody knows the reason, please advice me.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Bongjae Chang
>>>>
>>>>
>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>> To: <users_at_grizzly.dev.java.net>
>>>> Sent: Wednesday, June 17, 2009 6:05 PM
>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>> processing request
>>>>
>>>>
>>>>> Hi Abey,
>>>>>
>>>>> Thank you for testing it again.
>>>>>
>>>>> So now, maybe two issue exists.
>>>>>
>>>>> 1. processing request get stuck forever
>>>>> 2. Slow request issue
>>>>>
>>>>> Abey wrote:
>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>> delay in
>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>> I understood. Thank you for useful information.
>>>>>
>>>>> And I am trying to reproduce your issue continually.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> Bongjae Chang
>>>>>
>>>>>
>>>>> ----- Original Message ----- From: "Abey Tom"
>>>>> <appu_abey_tom_at_yahoo.com>
>>>>> To: <users_at_grizzly.dev.java.net>
>>>>> Sent: Wednesday, June 17, 2009 5:48 PM
>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>> processing request
>>>>>
>>>>>
>>>>>> I'm facing both stuck forever request(never released) and slow
>>>>>> request issues
>>>>>> even when I use the default thread pool configuration (min/max -
>>>>>> 5/5).
>>>>>> I tested with the scenarios with 1.
>>>>>> -Dcom.sun.grizzly.maxSelectorReadThread = 0 : The request
>>>>>> processing is
>>>>>> very quick but requests are getting stuck more often.
>>>>>> 2. -Dcom.sun.grizzly.maxSelectorReadThread = 1 or 2 : The request
>>>>>> processing is slower but some requests seems to get stuck but they
>>>>>> are
>>>>>> released. But STILL some requests are stuck forever.
>>>>>>
>>>>>> The chances of getting STUCK are MORE in Powerful server class
>>>>>> machines(4
>>>>>> core , 64bit, Linux) than in my laptop(Single core,32 bit ,
>>>>>> Windows ).
>>>>>>
>>>>>> Also abstract from first mail:
>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>> delay in
>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>
>>>>>> method DefaultThreadPool.execute(Runnable){ .... if (running){
>>>>>> try { Thread.sleep(2);
>>>>>> //or Thread.sleep(5);
>>>>>> } catch {....} workQueue.offer(task);
>>>>>> queueSize.incrementAndGet(); } }
>>>>>>
>>>>>> Thanks,
>>>>>> Abey Tom
>>>>>>
>>>>>>
>>>>>> BongJaeChang wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> When I tested it with various situations, I could meet some strange
>>>>>>>> behavior.
>>>>>>>>
>>>>>>>> When I used the read thread count which was not 0, processing
>>>>>>>> request was
>>>>>>>> very slow. For a while, it seemed getting stuck, but it always was
>>>>>>>> released.
>>>>>>>>
>>>>>>>> At first, I thought that the behavior was normal. But when I set
>>>>>>>> the read
>>>>>>>> thread count to be 0, I could know that processing request is
>>>>>>>> very quick.
>>>>>>>>
>>>>>>>> So, 1. Could you please test it again with
>>>>>>>> "-Dcom.sun.grizzly.maxSelectorReadThread=0" ?
>>>>>>>>
>>>>>>>> 2. And is your problem that processing request is slow or the
>>>>>>>> page gets
>>>>>>>> stuck and it is never released?
>>>>>>>>
>>>>>>>> It seems that it is not important the number of the read thread
>>>>>>>> count if
>>>>>>>> it is not 0 because I can always meet the symptom with setting
>>>>>>>> the read
>>>>>>>> thread count to be even 1.
>>>>>>>>
>>>>>>>> And the number of worker threads is not also important because
>>>>>>>> it can see
>>>>>>>> that the number has changed adequately. But it is not sure.
>>>>>>>>
>>>>>>>> So I am also debugging it in order to know why the page is slow
>>>>>>>> when I
>>>>>>>> set the read thread count to be even 1.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bongjae Chang
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/grizzly-comet-webserver-gets-stuck-sometime-while-processing-request-tp24033280p24069466.html
>>>>>>
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>> --------------------------------------------------------------------------------
>>>
>>>
>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>