Salut,
Bongjae Chang wrote:
> Hi Jeanfrancois,
>
> Maybe I think that I should explain more.
>
> When I reproduced and patched it, the key point was to prevent the StreamAlgorithm which was remained in HttpWorkerThread by old execution from being reused.
>
> Actually, it seemed that the problem was caused by StaticStreamAlgorithm's StaticHandler.
>
> As I attached the snapshots, whenever the problem was reproduced, the channel and response was not matched about only static contents.
>
> So I suspected StaticHandler's behavior.
>
> StaticHandler has a socketChannel and a fileCache. I think that sometimes remained socketChannel and fileCache caused unexpected result.
>
> Of course, I am not sure.
You are 100% right.
>
> But when I modified the HttpWorkerThread in order to clear the StaticHandler's fileCache and socketChannel, the problem was not reproduced.
>
> In AyncProtocolFilter#execute(), streamAlgorithm.setChannel() sets new SocketChannel and StaticHandler#attachChannel() is called.
>
> But it seems that it is not enough. When StaticHandler#handle() was called, could some timing problem be occurred? I am not sure.
You are right agaon
>
> But when I cleared the values, the problem just seemed to be resolved.
>
> So if you look the StaticHandler code, I think that you can find better solutions and the reason.
>
> I attached new patch. My attached patch is just a trial for reference and explaining my thought.
>
> Thanks!
>
> Index: com/sun/grizzly/http/HttpWorkerThread.java
> ===================================================================
> --- com/sun/grizzly/http/HttpWorkerThread.java (revision 3337)
> +++ com/sun/grizzly/http/HttpWorkerThread.java (working copy)
> @@ -43,6 +43,9 @@
>
> import com.sun.grizzly.util.StreamAlgorithm;
> import com.sun.grizzly.util.ThreadAttachment;
> +import com.sun.grizzly.util.Interceptor;
> +import com.sun.grizzly.standalone.StaticHandler;
> +
> import java.util.concurrent.ExecutorService;
>
> /**
> @@ -134,4 +137,15 @@
> }
> return threadAttachment;
> }
> +
> + @Override
> + public void reset() {
> + if( streamAlgorithm != null ) {
> + Interceptor handler = streamAlgorithm.getHandler();
> + if( handler instanceof StaticHandler ) {
> + ((StaticHandler) handler).reset();
> + }
> + }
> + super.reset();
> + }
> }
> Index: com/sun/grizzly/standalone/StaticHandler.java
> ===================================================================
> --- com/sun/grizzly/standalone/StaticHandler.java (revision 3337)
> +++ com/sun/grizzly/standalone/StaticHandler.java (working copy)
> @@ -160,4 +160,9 @@
> }
> return -1;
> }
> +
> + public void reset() {
> + fileCache = null;
> + socketChannel = null;
> + }
> }
Actually I like you patch. Go ahead and commit!
Many thanks!
-- Jeanfrancois
> --
> Bongjae Chang
>
>
> ----- Original Message -----
> From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
> To: <users_at_grizzly.dev.java.net>
> Sent: Thursday, June 18, 2009 9:51 AM
> Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
>
>
>> Salut,
>>
>> Bongjae Chang wrote:
>>> Hi Jeanfrancois,
>>>
>>> I tested issue #672("Setting readThreadCount > 1 significantly slow down") and it worked well.
>>>
>>> But Processing request still get stuck forever.
>> Yes since this is comet, this is expected (the connection gets suspended
>> until a server event happen on the server side). Did I understood you
>> properly?
>>
>> Thanks!
>>
>> -- Jeanfrancois
>>
>>
>>> Then,
>>>
>>> Bongjae wrote:
>>>> 1. processing request get stuck forever
>>>> 2. Slow request issue
>>> 1) should be fixed and 2) was resolved.
>>>
>>> Thanks!
>>> --
>>> Bongjae Chang
>>>
>>>
>>> ----- Original Message -----
>>> From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
>>> To: <users_at_grizzly.dev.java.net>
>>> Sent: Thursday, June 18, 2009 9:31 AM
>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while processing request
>>>
>>>
>>>> Salut,
>>>>
>>>> I think commit r3343 fixed the issue (at least I wasn't able to
>>>> reproduce it). Can you try and let me know!
>>>>
>>>> Thanks both of you for all the help! That was a nasty one!
>>>>
>>>> -- Jeanfrancois
>>>>
>>>> Jeanfrancois Arcand wrote:
>>>>> Salut,
>>>>>
>>>>> I've filled:
>>>>>
>>>>> https://grizzly.dev.java.net/issues/show_bug.cgi?id=672
>>>>>
>>>>> for the tracking the performance regression when readThread > 0
>>>>>
>>>>> A+
>>>>>
>>>>> -- Jeanfrancois
>>>>>
>>>>> Jeanfrancois Arcand wrote:
>>>>>> Salut,
>>>>>>
>>>>>> Great analysis as usual!
>>>>>>
>>>>>> if we set the processorTask to null for every request, we will regress
>>>>>> in term of performance as we will need to re-create/sync a new
>>>>>> instance. If DefaultProtocolFilter, we have:
>>>>>>
>>>>>>> 134 if (streamAlgorithm.parse(byteBuffer)){
>>>>>>> 135 136 if (processorTask == null){
>>>>>>> 137 processorTask = selectorThread.getProcessorTask();
>>>>>>> 138 workerThread.setProcessorTask(processorTask);
>>>>>>> 139 }
>>>>>> For normal processing, we set that value once and re-use it. For ARP
>>>>>> and Comet, we are using the c.s.g.arp.AsyncProtocolFilter, and we are
>>>>>> doing:
>>>>>>
>>>>>>> 162 if (streamAlgorithm.parse(byteBuffer)){
>>>>>>> 163 ProcessorTask processor =
>>>>>>> selectorThread.getProcessorTask();
>>>>>>> 164 configureProcessorTask(processor, ctx,
>>>>>>> 165 streamAlgorithm.getHandler(), inputStream);
>>>>>> we never get the ProcessorTask from the HttpWorkerThread. So I think
>>>>>> your patch fix the issue but it just a bug when readerThread > 1. I
>>>>>> suspect (I will take a look as I might be wrong) the
>>>>>> AsyncProtocolFilter is not added to the ProtocolChain because the ARP
>>>>>> is not enabled, hence we ends up with the DefaultProtocolFilter
>>>>>> instead of AsyncProtocolFilter.
>>>>>>
>>>>>> A+
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>>
>>>>>>
>>>>>> Bongjae Chang wrote:
>>>>>>> Hi Abey,
>>>>>>>
>>>>>>> I attached the proposed patch experimentally.
>>>>>>>
>>>>>>> I added HttpWorkerThread#reset() for cleaning up used thread.
>>>>>>>
>>>>>>> When I applied it, the problem never happened.
>>>>>>>
>>>>>>> Could you apply this patch and test it again?
>>>>>>>
>>>>>>> --
>>>>>>> Bongjae Chang
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>> Sent: Wednesday, June 17, 2009 8:45 PM
>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>> processing request
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> After all, I can reproduce it and I attach HttpWatch's snapshots.
>>>>>>>>
>>>>>>>> I found the machine which was reproduced. :)
>>>>>>>>
>>>>>>>> At first, I suspect DefaultThreadPool and FixedThreadPool. So I
>>>>>>>> tried to change the thread pool into JDKThreadPool.
>>>>>>>>
>>>>>>>> But this problem kept still.
>>>>>>>>
>>>>>>>> As you see the attched snapshot, whenever this problem is
>>>>>>>> reproduced, the http response is pushed.
>>>>>>>>
>>>>>>>> Here is example.
>>>>>>>>
>>>>>>>> In stuck.jpg,
>>>>>>>> ---
>>>>>>>> req: body-background.png
>>>>>>>> res: none
>>>>>>>>
>>>>>>>> req: main-background.png
>>>>>>>> res: body-background.png
>>>>>>>>
>>>>>>>> req: header-background.png
>>>>>>>> res: main-background.png
>>>>>>>> ---
>>>>>>>>
>>>>>>>> Maybe when request processing is very quick, it seems that
>>>>>>>> concurrent problem exists about responsing the http header and body.
>>>>>>>> But I am not sure.
>>>>>>>>
>>>>>>>> I am also curious to know how this phenomenon can happen and I will
>>>>>>>> try to analyze this as well curiously.
>>>>>>>>
>>>>>>>> But if anybody knows the reason, please advice me.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bongjae Chang
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message ----- From: "Bongjae Chang" <carryel_at_korea.com>
>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>> Sent: Wednesday, June 17, 2009 6:05 PM
>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>> processing request
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi Abey,
>>>>>>>>>
>>>>>>>>> Thank you for testing it again.
>>>>>>>>>
>>>>>>>>> So now, maybe two issue exists.
>>>>>>>>>
>>>>>>>>> 1. processing request get stuck forever
>>>>>>>>> 2. Slow request issue
>>>>>>>>>
>>>>>>>>> Abey wrote:
>>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>>> delay in
>>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>>> I understood. Thank you for useful information.
>>>>>>>>>
>>>>>>>>> And I am trying to reproduce your issue continually.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bongjae Chang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message ----- From: "Abey Tom"
>>>>>>>>> <appu_abey_tom_at_yahoo.com>
>>>>>>>>> To: <users_at_grizzly.dev.java.net>
>>>>>>>>> Sent: Wednesday, June 17, 2009 5:48 PM
>>>>>>>>> Subject: Re: grizzly-comet-webserver gets stuck sometime while
>>>>>>>>> processing request
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I'm facing both stuck forever request(never released) and slow
>>>>>>>>>> request issues
>>>>>>>>>> even when I use the default thread pool configuration (min/max -
>>>>>>>>>> 5/5).
>>>>>>>>>> I tested with the scenarios with 1.
>>>>>>>>>> -Dcom.sun.grizzly.maxSelectorReadThread = 0 : The request
>>>>>>>>>> processing is
>>>>>>>>>> very quick but requests are getting stuck more often.
>>>>>>>>>> 2. -Dcom.sun.grizzly.maxSelectorReadThread = 1 or 2 : The request
>>>>>>>>>> processing is slower but some requests seems to get stuck but they
>>>>>>>>>> are
>>>>>>>>>> released. But STILL some requests are stuck forever.
>>>>>>>>>>
>>>>>>>>>> The chances of getting STUCK are MORE in Powerful server class
>>>>>>>>>> machines(4
>>>>>>>>>> core , 64bit, Linux) than in my laptop(Single core,32 bit ,
>>>>>>>>>> Windows ).
>>>>>>>>>>
>>>>>>>>>> Also abstract from first mail:
>>>>>>>>>> One (strange??) thing I have noticed is that if I put a manual
>>>>>>>>>> delay in
>>>>>>>>>> processing (sleep) in as follows I'm not getting this issue.
>>>>>>>>>>
>>>>>>>>>> method DefaultThreadPool.execute(Runnable){ .... if (running){
>>>>>>>>>> try { Thread.sleep(2);
>>>>>>>>>> //or Thread.sleep(5);
>>>>>>>>>> } catch {....} workQueue.offer(task);
>>>>>>>>>> queueSize.incrementAndGet(); } }
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Abey Tom
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> BongJaeChang wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> When I tested it with various situations, I could meet some strange
>>>>>>>>>>>> behavior.
>>>>>>>>>>>>
>>>>>>>>>>>> When I used the read thread count which was not 0, processing
>>>>>>>>>>>> request was
>>>>>>>>>>>> very slow. For a while, it seemed getting stuck, but it always was
>>>>>>>>>>>> released.
>>>>>>>>>>>>
>>>>>>>>>>>> At first, I thought that the behavior was normal. But when I set
>>>>>>>>>>>> the read
>>>>>>>>>>>> thread count to be 0, I could know that processing request is
>>>>>>>>>>>> very quick.
>>>>>>>>>>>>
>>>>>>>>>>>> So, 1. Could you please test it again with
>>>>>>>>>>>> "-Dcom.sun.grizzly.maxSelectorReadThread=0" ?
>>>>>>>>>>>>
>>>>>>>>>>>> 2. And is your problem that processing request is slow or the
>>>>>>>>>>>> page gets
>>>>>>>>>>>> stuck and it is never released?
>>>>>>>>>>>>
>>>>>>>>>>>> It seems that it is not important the number of the read thread
>>>>>>>>>>>> count if
>>>>>>>>>>>> it is not 0 because I can always meet the symptom with setting
>>>>>>>>>>>> the read
>>>>>>>>>>>> thread count to be even 1.
>>>>>>>>>>>>
>>>>>>>>>>>> And the number of worker threads is not also important because
>>>>>>>>>>>> it can see
>>>>>>>>>>>> that the number has changed adequately. But it is not sure.
>>>>>>>>>>>>
>>>>>>>>>>>> So I am also debugging it in order to know why the page is slow
>>>>>>>>>>>> when I
>>>>>>>>>>>> set the read thread count to be even 1.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Bongjae Chang
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/grizzly-comet-webserver-gets-stuck-sometime-while-processing-request-tp24033280p24069466.html
>>>>>>>>>>
>>>>>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> --------------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net