Hi Jeanfrancois and Abey,
I fixed the issue at rev.3355.
Abey,
Could you please test and confirm it again? :)
If any problems remained about this issue, please let me know.
Thanks!
--
Bongjae Chang
----- Original Message -----
From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
To: <users_at_grizzly.dev.java.net>
Sent: Thursday, June 18, 2009 10:48 PM
Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
> Salut,
>
> Bongjae Chang wrote:
>> Hi Jeanfrancois,
>>
>> Maybe I think that I should explain more.
>>
>> When I reproduced and patched it, the key point was to prevent the StreamAlgorithm which was remained in HttpWorkerThread by old execution from being reused.
>>
>> Actually, it seemed that the problem was caused by StaticStreamAlgorithm's StaticHandler.
>>
>> As I attached the snapshots, whenever the problem was reproduced, the channel and response was not matched about only static contents.
>>
>> So I suspected StaticHandler's behavior.
>>
>> StaticHandler has a socketChannel and a fileCache. I think that sometimes remained socketChannel and fileCache caused unexpected result.
>>
>> Of course, I am not sure.
>
> You are 100% right.
>
>>
>> But when I modified the HttpWorkerThread in order to clear the StaticHandler's fileCache and socketChannel, the problem was not reproduced.
>>
>> In AyncProtocolFilter#execute(), streamAlgorithm.setChannel() sets new SocketChannel and StaticHandler#attachChannel() is called.
>>
>> But it seems that it is not enough. When StaticHandler#handle() was called, could some timing problem be occurred? I am not sure.
>
> You are right agaon
>
>
>>
>> But when I cleared the values, the problem just seemed to be resolved.
>>
>> So if you look the StaticHandler code, I think that you can find better solutions and the reason.
>>
>> I attached new patch. My attached patch is just a trial for reference and explaining my thought.
>>
>> Thanks!
>>
>> Index: com/sun/grizzly/http/HttpWorkerThread.java
>> ===================================================================
>> --- com/sun/grizzly/http/HttpWorkerThread.java (revision 3337)
>> +++ com/sun/grizzly/http/HttpWorkerThread.java (working copy)
>> @@ -43,6 +43,9 @@
>>
>> import com.sun.grizzly.util.StreamAlgorithm;
>> import com.sun.grizzly.util.ThreadAttachment;
>> +import com.sun.grizzly.util.Interceptor;
>> +import com.sun.grizzly.standalone.StaticHandler;
>> +
>> import java.util.concurrent.ExecutorService;
>>
>> /**
>> @@ -134,4 +137,15 @@
>> }
>> return threadAttachment;
>> }
>> +
>> + @Override
>> + public void reset() {
>> + if( streamAlgorithm != null ) {
>> + Interceptor handler = streamAlgorithm.getHandler();
>> + if( handler instanceof StaticHandler ) {
>> + ((StaticHandler) handler).reset();
>> + }
>> + }
>> + super.reset();
>> + }
>> }
>> Index: com/sun/grizzly/standalone/StaticHandler.java
>> ===================================================================
>> --- com/sun/grizzly/standalone/StaticHandler.java (revision 3337)
>> +++ com/sun/grizzly/standalone/StaticHandler.java (working copy)
>> @@ -160,4 +160,9 @@
>> }
>> return -1;
>> }
>> +
>> + public void reset() {
>> + fileCache = null;
>> + socketChannel = null;
>> + }
>> }
>
> Actually I like you patch. Go ahead and commit!
>
> Many thanks!
>
> -- Jeanfrancois
>
>
>
>
>> --
>> Bongjae Chang
>>
>>
>> ----- Original Message -----
>> From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
>> To: <users_at_grizzly.dev.java.net>
>> Sent: Thursday, June 18, 2009 9:51 AM
>> Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
>>
>>
>>> Salut,
>>>
>>> Bongjae Chang wrote:
>>>> Hi Jeanfrancois,
>>>>
>>>> I tested issue #672("Setting readThreadCount > 1 significantly slow down") and it worked well.
>>>>
>>>> But Processing request still get stuck forever.
>>> Yes since this is comet, this is expected (the connection gets suspended
>>> until a server event happen on the server side). Did I understood you
>>> properly?
>>>
>>> Thanks!
>>>
>>> -- Jeanfrancois
>>>
>>>
>>>> Then,
>>>>
>>>> Bongjae wrote:
>>>>> 1. processing request get stuck forever
>>>>> 2. Slow request issue
>>>> 1) should be fixed and 2) was resolved.
>>>>
>>>> Thanks!
>>>> --
>>>> Bongjae Chang
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
>>>> To: <users_at_grizzly.dev.java.net>
>>>> Sent: Thursday, June 18, 2009 9:31 AM
>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
>>>>
>>>>
>>>>> Salut,
>>>>>
>>>>> I think commit r3343 fixed the issue (at least I wasn't able to
>>>>> reproduce it). Can you try and let me know!
>>>>>
>>>>> Thanks both of you for all the help! That was a nasty one!
>>>>>
>>>>> -- Jeanfrancois
>>>>>
>>>>> Jeanfrancois Arcand wrote:
>>>>>> Salut,
>>>>>>
>>>>>> I've filled:
>>>>>>
>>>>>> https://grizzly.dev.java.net/issues/show_bug.cgi?id=672
>>>>>>
>>>>>> for the tracking the performance regression when readThread > 0
>>>>>>
>>>>>> A+
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>> Jeanfrancois Arcand wrote:
>>>>>>> Salut,
>>>>>>>
>>>>>>> Great analysis as usual!
>>>>>>>
>>>>>>> if we set the processorTask to null for every request, we will regress
>>>>>>> in term of performance as we will need to re-create/sync a new
>>>>>>> instance. If DefaultProtocolFilter, we have:
>>>>>>>
>>>>>>>> 134 if (streamAlgorithm.parse(byteBuffer)){
>>>>>>>> 135 136 if (processorTask == null){
>>>>>>>> 137 processorTask = selectorThread.getProcessorTask();
>>>>>>>> 138 workerThread.setProcessorTask(processorTask);
>>>>>>>> 139 }
>>>>>>> For normal processing, we set that value once and re-use it. For ARP
>>>>>>> and Comet, we are using the c.s.g.arp.AsyncProtocolFilter, and we are
>>>>>>> doing:
>>>>>>>
>>>>>>>> 162 if (streamAlgorithm.parse(byteBuffer)){
>>>>>>>> 163 ProcessorTask processor =
>>>>>>>> selectorThread.getProcessorTask();
>>>>>>>> 164 configureProcessorTask(processor, ctx,
>>>>>>>> 165 streamAlgorithm.getHandler(), inputStream);
>>>>>>> we never get the ProcessorTask from the HttpWorkerThread. So I think
>>>>>>> your patch fix the issue but it just a bug when readerThread > 1. I
>>>>>>> suspect (I will take a look as I might be wrong) the
>>>>>>> AsyncProtocolFilter is not added to the ProtocolChain because the ARP
>>>>>>> is not enabled, hence we ends up with the DefaultProtocolFilter
>>>>>>> instead of AsyncProtocolFilter.
>>>>>>>
>>>>>>> A+
>>>>>>>
>>>>>>> -- Jeanfrancois
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Bongjae Chang wrote:
>>>>>>>> Hi Abey,
>>>>>>>>
>>>>>>>> I attached the proposed patch experimentally.
>>>>>>>>
>>>>>>>> I added HttpWorkerThread#reset() for cleaning up used thread.
>>>>>>>>
>>>>>>>> When I applied it, the problem never happened.
>>>>>>>>
>>>>>>>> Could you apply this patch and test it again?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bongjae Chang
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>> Sent: Wednesday, June 17, 2009 8:45 PM
>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>> processing request
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> After all, I can reproduce it and I attach HttpWatch's snapshots.
>>>>>>>>>
>>>>>>>>> I found the machine which was reproduced. :)
>>>>>>>>>
>>>>>>>>> At first, I suspect DefaultThreadPool and FixedThreadPool. So I
>>>>>>>>> tried to change the thread pool into JDKThreadPool.
>>>>>>>>>
>>>>>>>>> But this problem kept still.
>>>>>>>>>
>>>>>>>>> As you see the attched snapshot, whenever this problem is
>>>>>>>>> reproduced, the http response is pushed.
>>>>>>>>>
>>>>>>>>> Here is example.
>>>>>>>>>
>>>>>>>>> In stuck.jpg,
>>>>>>>>> ---
>>>>>>>>> req: body-background.png
>>>>>>>>> res: none
>>>>>>>>>
>>>>>>>>> req: main-background.png
>>>>>>>>> res: body-background.png
>>>>>>>>>
>>>>>>>>> req: header-background.png
>>>>>>>>> res: main-background.png
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> Maybe when request processing is very quick, it seems that
>>>>>>>>> concurrent problem exists about responsing the http header and body.
>>>>>>>>> But I am not sure.
>>>>>>>>>
>>>>>>>>> I am also curious to know how this phenomenon can happen and I will
>>>>>>>>> try to analyze this as well curiously.
>>>>>>>>>
>>>>>>>>> But if anybody knows the reason, please advice me.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bongjae Chang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>>> Sent: Wednesday, June 17, 2009 6:05 PM
>>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>>> processing request
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Abey,
>>>>>>>>>>
>>>>>>>>>> Thank you for testing it again.
>>>>>>>>>>
>>>>>>>>>> So now, maybe two issue exists.
>>>>>>>>>>
>>>>>>>>>> 1. processing request get stuck forever
>>>>>>>>>> 2. Slow request issue
>>>>>>>>>>
>>>>>>>>>> Abey wrote:
>>>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>>>> delay in
>>>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>>>> I understood. Thank you for useful information.
>>>>>>>>>>
>>>>>>>>>> And I am trying to reproduce your issue continually.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Bongjae Chang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message ----- From: "Abey Tom"
>>>>>>>>>> <appu_abey_tom_at_yahoo.com>
>>>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>>>> Sent: Wednesday, June 17, 2009 5:48 PM
>>>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>>>> processing request
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I'm facing both stuck forever request(never released) and slow
>>>>>>>>>>> request issues
>>>>>>>>>>> even when I use the default thread pool configuration (min/max -
>>>>>>>>>>> 5/5).
>>>>>>>>>>> I tested with the scenarios with 1.
>>>>>>>>>>> -Dcom.sun.grizzly.maxSelectorReadThread = 0 : The request
>>>>>>>>>>> processing is
>>>>>>>>>>> very quick but requests are getting stuck more often.
>>>>>>>>>>> 2. -Dcom.sun.grizzly.maxSelectorReadThread = 1 or 2 : The request
>>>>>>>>>>> processing is slower but some requests seems to get stuck but they
>>>>>>>>>>> are
>>>>>>>>>>> released. But STILL some requests are stuck forever.
>>>>>>>>>>>
>>>>>>>>>>> The chances of getting STUCK are MORE in Powerful server class
>>>>>>>>>>> machines(4
>>>>>>>>>>> core , 64bit, Linux) than in my laptop(Single core,32 bit ,
>>>>>>>>>>> Windows ).
>>>>>>>>>>>
>>>>>>>>>>> Also abstract from first mail:
>>>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>>>> delay in
>>>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>>>>>
>>>>>>>>>>> method DefaultThreadPool.execute(Runnable){ .... if (running){
>>>>>>>>>>> try { Thread.sleep(2);
>>>>>>>>>>> //or Thread.sleep(5);
>>>>>>>>>>> } catch {....} workQueue.offer(task);
>>>>>>>>>>> queueSize.incrementAndGet(); } }
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Abey Tom
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> BongJaeChang wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I tested it with various situations, I could meet some strange
>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I used the read thread count which was not 0, processing
>>>>>>>>>>>>> request was
>>>>>>>>>>>>> very slow. For a while, it seemed getting stuck, but it always was
>>>>>>>>>>>>> released.
>>>>>>>>>>>>>
>>>>>>>>>>>>> At first, I thought that the behavior was normal. But when I set
>>>>>>>>>>>>> the read
>>>>>>>>>>>>> thread count to be 0, I could know that processing request is
>>>>>>>>>>>>> very quick.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, 1. Could you please test it again with
>>>>>>>>>>>>> "-Dcom.sun.grizzly.maxSelectorReadThread=0" ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. And is your problem that processing request is slow or the
>>>>>>>>>>>>> page gets
>>>>>>>>>>>>> stuck and it is never released?
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems that it is not important the number of the read thread
>>>>>>>>>>>>> count if
>>>>>>>>>>>>> it is not 0 because I can always meet the symptom with setting
>>>>>>>>>>>>> the read
>>>>>>>>>>>>> thread count to be even 1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And the number of worker threads is not also important because
>>>>>>>>>>>>> it can see
>>>>>>>>>>>>> that the number has changed adequately. But it is not sure.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I am also debugging it in order to know why the page is slow
>>>>>>>>>>>>> when I
>>>>>>>>>>>>> set the read thread count to be even 1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Bongjae Chang
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://www.nabble.com/grizzly-comet-webserver-gets-stuck-sometime-while-processing-request-tp24033280p24069466.html
>>>>>>>>>>>
>>>>>>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
>