Hi Jeanfrancois,
Maybe I think that I should explain more.
When I reproduced and patched it, the key point was to prevent the StreamAlgorithm which was remained in HttpWorkerThread by old execution from being reused.
Actually, it seemed that the problem was caused by StaticStreamAlgorithm's StaticHandler.
As I attached the snapshots, whenever the problem was reproduced, the channel and response was not matched about only static contents.
So I suspected StaticHandler's behavior.
StaticHandler has a socketChannel and a fileCache. I think that sometimes remained socketChannel and fileCache caused unexpected result.
Of course, I am not sure.
But when I modified the HttpWorkerThread in order to clear the StaticHandler's fileCache and socketChannel, the problem was not reproduced.
In AyncProtocolFilter#execute(), streamAlgorithm.setChannel() sets new SocketChannel and StaticHandler#attachChannel() is called.
But it seems that it is not enough. When StaticHandler#handle() was called, could some timing problem be occurred? I am not sure.
But when I cleared the values, the problem just seemed to be resolved.
So if you look the StaticHandler code, I think that you can find better solutions and the reason.
I attached new patch. My attached patch is just a trial for reference and explaining my thought.
Thanks!
Index: com/sun/grizzly/http/HttpWorkerThread.java
===================================================================
--- com/sun/grizzly/http/HttpWorkerThread.java (revision 3337)
+++ com/sun/grizzly/http/HttpWorkerThread.java (working copy)
@@ -43,6 +43,9 @@
import com.sun.grizzly.util.StreamAlgorithm;
import com.sun.grizzly.util.ThreadAttachment;
+import com.sun.grizzly.util.Interceptor;
+import com.sun.grizzly.standalone.StaticHandler;
+
import java.util.concurrent.ExecutorService;
/**
@@ -134,4 +137,15 @@
}
return threadAttachment;
}
+
+ @Override
+ public void reset() {
+ if( streamAlgorithm != null ) {
+ Interceptor handler = streamAlgorithm.getHandler();
+ if( handler instanceof StaticHandler ) {
+ ((StaticHandler) handler).reset();
+ }
+ }
+ super.reset();
+ }
}
Index: com/sun/grizzly/standalone/StaticHandler.java
===================================================================
--- com/sun/grizzly/standalone/StaticHandler.java (revision 3337)
+++ com/sun/grizzly/standalone/StaticHandler.java (working copy)
@@ -160,4 +160,9 @@
}
return -1;
}
+
+ public void reset() {
+ fileCache = null;
+ socketChannel = null;
+ }
}
--
Bongjae Chang
----- Original Message -----
From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
To: <users_at_grizzly.dev.java.net>
Sent: Thursday, June 18, 2009 9:51 AM
Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
> Salut,
>
> Bongjae Chang wrote:
>> Hi Jeanfrancois,
>>
>> I tested issue #672("Setting readThreadCount > 1 significantly slow down") and it worked well.
>>
>> But Processing request still get stuck forever.
>
> Yes since this is comet, this is expected (the connection gets suspended
> until a server event happen on the server side). Did I understood you
> properly?
>
> Thanks!
>
> -- Jeanfrancois
>
>
>>
>> Then,
>>
>> Bongjae wrote:
>>> 1. processing request get stuck forever
>>> 2. Slow request issue
>>
>> 1) should be fixed and 2) was resolved.
>>
>> Thanks!
>> --
>> Bongjae Chang
>>
>>
>> ----- Original Message -----
>> From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
>> To: <users_at_grizzly.dev.java.net>
>> Sent: Thursday, June 18, 2009 9:31 AM
>> Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
>>
>>
>>> Salut,
>>>
>>> I think commit r3343 fixed the issue (at least I wasn't able to
>>> reproduce it). Can you try and let me know!
>>>
>>> Thanks both of you for all the help! That was a nasty one!
>>>
>>> -- Jeanfrancois
>>>
>>> Jeanfrancois Arcand wrote:
>>>> Salut,
>>>>
>>>> I've filled:
>>>>
>>>> https://grizzly.dev.java.net/issues/show_bug.cgi?id=672
>>>>
>>>> for the tracking the performance regression when readThread > 0
>>>>
>>>> A+
>>>>
>>>> -- Jeanfrancois
>>>>
>>>> Jeanfrancois Arcand wrote:
>>>>> Salut,
>>>>>
>>>>> Great analysis as usual!
>>>>>
>>>>> if we set the processorTask to null for every request, we will regress
>>>>> in term of performance as we will need to re-create/sync a new
>>>>> instance. If DefaultProtocolFilter, we have:
>>>>>
>>>>>> 134 if (streamAlgorithm.parse(byteBuffer)){
>>>>>> 135 136 if (processorTask == null){
>>>>>> 137 processorTask = selectorThread.getProcessorTask();
>>>>>> 138 workerThread.setProcessorTask(processorTask);
>>>>>> 139 }
>>>>> For normal processing, we set that value once and re-use it. For ARP
>>>>> and Comet, we are using the c.s.g.arp.AsyncProtocolFilter, and we are
>>>>> doing:
>>>>>
>>>>>> 162 if (streamAlgorithm.parse(byteBuffer)){
>>>>>> 163 ProcessorTask processor =
>>>>>> selectorThread.getProcessorTask();
>>>>>> 164 configureProcessorTask(processor, ctx,
>>>>>> 165 streamAlgorithm.getHandler(), inputStream);
>>>>> we never get the ProcessorTask from the HttpWorkerThread. So I think
>>>>> your patch fix the issue but it just a bug when readerThread > 1. I
>>>>> suspect (I will take a look as I might be wrong) the
>>>>> AsyncProtocolFilter is not added to the ProtocolChain because the ARP
>>>>> is not enabled, hence we ends up with the DefaultProtocolFilter
>>>>> instead of AsyncProtocolFilter.
>>>>>
>>>>> A+
>>>>>
>>>>> -- Jeanfrancois
>>>>>
>>>>>
>>>>>
>>>>> Bongjae Chang wrote:
>>>>>> Hi Abey,
>>>>>>
>>>>>> I attached the proposed patch experimentally.
>>>>>>
>>>>>> I added HttpWorkerThread#reset() for cleaning up used thread.
>>>>>>
>>>>>> When I applied it, the problem never happened.
>>>>>>
>>>>>> Could you apply this patch and test it again?
>>>>>>
>>>>>> --
>>>>>> Bongjae Chang
>>>>>>
>>>>>>
>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>> Sent: Wednesday, June 17, 2009 8:45 PM
>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>> processing request
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> After all, I can reproduce it and I attach HttpWatch's snapshots.
>>>>>>>
>>>>>>> I found the machine which was reproduced. :)
>>>>>>>
>>>>>>> At first, I suspect DefaultThreadPool and FixedThreadPool. So I
>>>>>>> tried to change the thread pool into JDKThreadPool.
>>>>>>>
>>>>>>> But this problem kept still.
>>>>>>>
>>>>>>> As you see the attched snapshot, whenever this problem is
>>>>>>> reproduced, the http response is pushed.
>>>>>>>
>>>>>>> Here is example.
>>>>>>>
>>>>>>> In stuck.jpg,
>>>>>>> ---
>>>>>>> req: body-background.png
>>>>>>> res: none
>>>>>>>
>>>>>>> req: main-background.png
>>>>>>> res: body-background.png
>>>>>>>
>>>>>>> req: header-background.png
>>>>>>> res: main-background.png
>>>>>>> ---
>>>>>>>
>>>>>>> Maybe when request processing is very quick, it seems that
>>>>>>> concurrent problem exists about responsing the http header and body.
>>>>>>> But I am not sure.
>>>>>>>
>>>>>>> I am also curious to know how this phenomenon can happen and I will
>>>>>>> try to analyze this as well curiously.
>>>>>>>
>>>>>>> But if anybody knows the reason, please advice me.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Bongjae Chang
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>> Sent: Wednesday, June 17, 2009 6:05 PM
>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>> processing request
>>>>>>>
>>>>>>>
>>>>>>>> Hi Abey,
>>>>>>>>
>>>>>>>> Thank you for testing it again.
>>>>>>>>
>>>>>>>> So now, maybe two issue exists.
>>>>>>>>
>>>>>>>> 1. processing request get stuck forever
>>>>>>>> 2. Slow request issue
>>>>>>>>
>>>>>>>> Abey wrote:
>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>> delay in
>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>> I understood. Thank you for useful information.
>>>>>>>>
>>>>>>>> And I am trying to reproduce your issue continually.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bongjae Chang
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message ----- From: "Abey Tom"
>>>>>>>> <appu_abey_tom_at_yahoo.com>
>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>> Sent: Wednesday, June 17, 2009 5:48 PM
>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>> processing request
>>>>>>>>
>>>>>>>>
>>>>>>>>> I'm facing both stuck forever request(never released) and slow
>>>>>>>>> request issues
>>>>>>>>> even when I use the default thread pool configuration (min/max -
>>>>>>>>> 5/5).
>>>>>>>>> I tested with the scenarios with 1.
>>>>>>>>> -Dcom.sun.grizzly.maxSelectorReadThread = 0 : The request
>>>>>>>>> processing is
>>>>>>>>> very quick but requests are getting stuck more often.
>>>>>>>>> 2. -Dcom.sun.grizzly.maxSelectorReadThread = 1 or 2 : The request
>>>>>>>>> processing is slower but some requests seems to get stuck but they
>>>>>>>>> are
>>>>>>>>> released. But STILL some requests are stuck forever.
>>>>>>>>>
>>>>>>>>> The chances of getting STUCK are MORE in Powerful server class
>>>>>>>>> machines(4
>>>>>>>>> core , 64bit, Linux) than in my laptop(Single core,32 bit ,
>>>>>>>>> Windows ).
>>>>>>>>>
>>>>>>>>> Also abstract from first mail:
>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>> delay in
>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>>>
>>>>>>>>> method DefaultThreadPool.execute(Runnable){ .... if (running){
>>>>>>>>> try { Thread.sleep(2);
>>>>>>>>> //or Thread.sleep(5);
>>>>>>>>> } catch {....} workQueue.offer(task);
>>>>>>>>> queueSize.incrementAndGet(); } }
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Abey Tom
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> BongJaeChang wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> When I tested it with various situations, I could meet some strange
>>>>>>>>>>> behavior.
>>>>>>>>>>>
>>>>>>>>>>> When I used the read thread count which was not 0, processing
>>>>>>>>>>> request was
>>>>>>>>>>> very slow. For a while, it seemed getting stuck, but it always was
>>>>>>>>>>> released.
>>>>>>>>>>>
>>>>>>>>>>> At first, I thought that the behavior was normal. But when I set
>>>>>>>>>>> the read
>>>>>>>>>>> thread count to be 0, I could know that processing request is
>>>>>>>>>>> very quick.
>>>>>>>>>>>
>>>>>>>>>>> So, 1. Could you please test it again with
>>>>>>>>>>> "-Dcom.sun.grizzly.maxSelectorReadThread=0" ?
>>>>>>>>>>>
>>>>>>>>>>> 2. And is your problem that processing request is slow or the
>>>>>>>>>>> page gets
>>>>>>>>>>> stuck and it is never released?
>>>>>>>>>>>
>>>>>>>>>>> It seems that it is not important the number of the read thread
>>>>>>>>>>> count if
>>>>>>>>>>> it is not 0 because I can always meet the symptom with setting
>>>>>>>>>>> the read
>>>>>>>>>>> thread count to be even 1.
>>>>>>>>>>>
>>>>>>>>>>> And the number of worker threads is not also important because
>>>>>>>>>>> it can see
>>>>>>>>>>> that the number has changed adequately. But it is not sure.
>>>>>>>>>>>
>>>>>>>>>>> So I am also debugging it in order to know why the page is slow
>>>>>>>>>>> when I
>>>>>>>>>>> set the read thread count to be even 1.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bongjae Chang
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/grizzly-comet-webserver-gets-stuck-sometime-while-processing-request-tp24033280p24069466.html
>>>>>>>>>
>>>>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --------------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
>