I've commited workaround for selector spin issue.
Basically fix contains of 2 parts:
1) detecting empty selector spin
2) perform workaround
In order to detect empty selector spin - we introduce 2 threshold
values: select time threshold and empty spin count threshold.
Each time selector.select() returns empty key set - we check the time
selector.select() took; if the time is less than "select time
threshold" - we increment the empty spin counter. Once empty spin
counter reaches "empty spin count threshold" - we assume that empty
selector spin is detected - and perform workaround, which means we
create new Selector and reregister all the keys from "bad" selector to
new one. Just want to add that each successful selector.select()
resets empty spin counter.
If you have any feedback - let us know.
And thank you, John, for cooperation!!!
WBR,
Alexey.
[1] Author: oleksiys
Date: 2009-04-22 09:59:29+0000
New Revision: 3032
Modified:
trunk/code/modules/grizzly/src/main/java/com/sun/grizzly/
Controller.java
trunk/code/modules/grizzly/src/main/java/com/sun/grizzly/
SelectorHandler.java
trunk/code/modules/grizzly/src/main/java/com/sun/grizzly/
TCPSelectorHandler.java
On Apr 18, 2009, at 13:04 , John Franey wrote:
>
>
> On Fri, Apr 17, 2009 at 8:21 AM, Oleksiy Stashok <Oleksiy.Stashok_at_sun.com
> > wrote:
>>>
>>>
>>> Apr 16, 2009 9:29:22 PM com.sun.grizzly.Controller
>>> handleSelectException
>>> SEVERE: Selector was unexpectedly closed.
>>> java.io.IOException: File exists
>>> at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>> at
>>> sun
>>> .nio
>>> .ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:
>>> 233)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:214)
>>> at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:
>>> 528)
>>> at com.sun.grizzly.Controller.doSelect(Controller.java:328)
>>> at
>>> com
>>> .sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
>>> 90)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> Thanks for looking and for asking. I had to modify
>>> Controller.java to print the stack trace in this log message.
>>> Until now I didn't know I was getting the File exists exception,
>>> just as in the bug report I quoted above.
>> Thank you. I've fixed Controller code to show SEVERE exceptions
>> stacktrace.
>>
>>
>> Ok, so do you need me to file an issue report?
> Yes, and if you have workaround - will appreciate if you can share it.
>
> Thanks.
>
> WBR,
> Alexey.
>
> Alexey,
>
> I don't have a workaround to the cpu spin on select. I'm hoping to
> find one here.
>
> From my first entry on this thread:
>
> Please see especially, this comment from Emannual Lecharmy (31/Mar/
> 09 03:23 PM).
>
> So, I'm looking for this 'beautiful hack' in the Controller source
> code for 1.9.11 and am not seeing it. I think maybe I'm blind :) or
> the hack is in grizzly 2.0? If its in 2.0, do you think I can make
> it fit into 1.9.11? Or do you advise I try my application on 2.0
> trunk?
>
> Jeanfrancois said:
> I didn't put the hack there (just for Comet) as I never saw it on
> the Controller. Can you share a test case?
>
> I am not having success finding this hack in Comet. Any pointer?
>
> Thanks
>
>
>>
>>
>> Thanks,
>> John
>>
>>
>>>
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> Thanks.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>>
>>>>
>>>> Regards,
>>>> John
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>> John
>>>>
>>>>
>>>>
>>>> On Tue, Apr 14, 2009 at 10:14 AM, Jeanfrancois Arcand <Jeanfrancois.Arcand_at_sun.com
>>>> <mailto:Jeanfrancois.Arcand_at_sun.com>> wrote:
>>>>
>>>> Salut,
>>>>
>>>> back online
>>>>
>>>>
>>>> John Franey wrote:
>>>>
>>>> Jeanfrancois,
>>>>
>>>> Just a couple of clarifications:
>>>>
>>>> 1) Running from the command line needs the jar files too.
>>>> Sorry
>>>> I gave the impression that it was too easy. I think its
>>>> easiest
>>>> to run from IDE, especially since a debugger is needed. I
>>>> hope
>>>> that is ok.
>>>>
>>>>
>>>> OK I will try to reproduce the issue but I usually don't use
>>>> IDE (I
>>>> use vim :-)).
>>>>
>>>>
>>>>
>>>>
>>>> 2) Also, the BReceiver won't be idle during the rest sleeps.
>>>> Sorry about that. Instead, once the exception occurs,
>>>> simply
>>>> kill the UDPSender process, then you will be sure the
>>>> BReceiver
>>>> is not receiving any new packets.
>>>>
>>>>
>>>> OK. BTW I've talked to the NIO/JDK lead and nothing about
>>>> spinning
>>>> selector has been reported on jdk 4/5/6 so we might need to
>>>> closely
>>>> look at your code.
>>>>
>>>> Thanks
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>>
>>>> Regards,
>>>> John
>>>>
>>>>
>>>>
>>>> On Thu, Apr 9, 2009 at 2:31 PM, John Franey <jjfraney_at_gmail.com
>>>> <mailto:jjfraney_at_gmail.com> <mailto:jjfraney_at_gmail.com
>>>>
>>>> <mailto:jjfraney_at_gmail.com>>> wrote:
>>>>
>>>> Jeanfrancois,
>>>>
>>>> I got lucky, here is a test case. (gmail didn't let me
>>>> send
>>>> it as
>>>> zip, so its a jar).
>>>>
>>>> this is a maven project, I guess you can build and run.
>>>>
>>>> There are two main classes.
>>>>
>>>> A receiver, BReceiver, and a sender, UDPSender.
>>>>
>>>> I run these from eclipse. I guess your IDE will let
>>>> your run
>>>> them
>>>> easily, but I guess from the command line something
>>>> easy like:
>>>> java -cp target/classes grizzlyproto/BReceiver would run.
>>>>
>>>> BReciever cycles a single udp port through many
>>>> connections.
>>>> UDPSender sends datagrams to that port at high rate.
>>>>
>>>> Periodically, BReceiver pauses at a time when there
>>>> should be
>>>> no cpu
>>>> charges to that process. Not sure this helps, but the
>>>> intention was
>>>> that if the process is using cpu at this time, then the
>>>> selector is
>>>> spinning.
>>>>
>>>> Within a few moment, BReceiver will get an exception (I
>>>> do
>>>> not know
>>>> if the exception is related to the problem). Then you
>>>> can set
>>>> breakpoint on Controller.doSelect's return from
>>>> selectorHandler.select (at line 329) to inspect value of
>>>> readyKeys. Its empty.
>>>>
>>>>
>>>> Here are my system specificaitons (from ubuntu 8.10)
>>>> Linux isidore 2.6.27-11-generic #1 SMP Thu Jan 29
>>>> 19:28:32
>>>> UTC 2009
>>>> x86_64 GNU/Linux
>>>>
>>>>
>>>> I get the same behavior on both of these:
>>>>
>>>> java version "1.6.0_12"
>>>> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01,
>>>> mixed mode)
>>>>
>>>> and
>>>>
>>>> java version "1.6.0_03"
>>>> Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
>>>> BEA JRockit(R) (build
>>>> R27.5.0-110-94909-1.6.0_03-20080204-1558-linux-x86_64,
>>>> compiled mode)
>>>>
>>>>
>>>> Regards,
>>>> John
>>>>
>>>>
>>>>
>>>> On Thu, Apr 9, 2009 at 2:00 PM, John Franey
>>>> <jjfraney_at_gmail.com <mailto:jjfraney_at_gmail.com>
>>>> <mailto:jjfraney_at_gmail.com
>>>> <mailto:jjfraney_at_gmail.com>>> wrote:
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Thu, Apr 9, 2009 at 1:02 PM, Jeanfrancois Arcand
>>>> <Jeanfrancois.Arcand_at_sun.com
>>>> <mailto:Jeanfrancois.Arcand_at_sun.com>
>>>> <mailto:Jeanfrancois.Arcand_at_sun.com
>>>> <mailto:Jeanfrancois.Arcand_at_sun.com>>> wrote:
>>>>
>>>> Salut,
>>>>
>>>>
>>>> John Franey wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm running stress tests on linux. At some
>>>> point, cpu
>>>> utilization goes high. I've verified using
>>>> the
>>>> debugger
>>>> that Controller.doSelect gets zero keys
>>>> back from
>>>> handler's select.
>>>>
>>>>
>>>> Yes this is an issue I've reported awhile ago
>>>> to the JDK
>>>> team. This sounds like bugparade 6403933
>>>>
>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933
>>>>
>>>> A connection is reset causes the file
>>>> descriptor to be
>>>> selected with events POLLHUP and POLLERR but
>>>> unless
>>>> there is
>>>> data waiting to be read then channel will not be
>>>> selected.
>>>> It's a awkward problem.
>>>>
>>>> This has been fixed in JDK 7 build 50 and will be
>>>> included
>>>> in JDK 6u16.
>>>>
>>>>
>>>> The last statement in the evaluation of this bug
>>>> nails
>>>> it, and
>>>> the load pattern described in this bug matches
>>>> mine: http://bugs.sun.com/view_bug.do?bug_id=6693490
>>>>
>>>> It sounds like the same root cause?
>>>>
>>>>
>>>>
>>>> Google search led me to this page at apache
>>>> mina jira
>>>> DIRMINA-678
>>>> <https://issues.apache.org/jira/browse/DIRMINA-678
>>>> >.
>>>>
>>>> Please see especially, this comment from
>>>> Emannual
>>>> Lecharmy (31/Mar/09 03:23 PM)
>>>> <https://issues.apache.org/jira/browse/DIRMINA-678?focusedCommentId=12696711&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
>>>> #action_12696711
>>>> <https://issues.apache.org/jira/browse/DIRMINA-678?focusedCommentId=12696711&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12696711
>>>> >
>>>> <https://issues.apache.org/jira/browse/DIRMINA-678?focusedCommentId=12696711&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
>>>> #action_12696711
>>>> <https://issues.apache.org/jira/browse/DIRMINA-678?focusedCommentId=12696711&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12696711
>>>> >>>.
>>>>
>>>>
>>>> So, I'm looking for this 'beautiful hack'
>>>> in the
>>>> Controller source code for 1.9.11 and am not
>>>> seeing it.
>>>> I think maybe I'm blind :) or the hack is
>>>> in grizzly
>>>> 2.0? If its in 2.0, do you think I can
>>>> make it
>>>> fit into
>>>> 1.9.11? Or do you advise I try my
>>>> application on
>>>> 2.0 trunk?
>>>>
>>>>
>>>> I didn't put the hack there (just for Comet) as I
>>>> never saw
>>>> it on the Controller. Can you share a test case?
>>>>
>>>>
>>>> I don't have a test case and am working on one. I
>>>> would be
>>>> surprised if I can get it to happen without a lot
>>>> of work.
>>>>
>>>> Is there a code sample from comet that has this
>>>> hack in it to
>>>> see if I can make the same change for my app?
>>>>
>>>> Thanks
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> users-unsubscribe_at_grizzly.dev.java.net
>>>> <mailto:users-unsubscribe_at_grizzly.dev.java.net>
>>>> <mailto:users-unsubscribe_at_grizzly.dev.java.net
>>>> <mailto:users-unsubscribe_at_grizzly.dev.java.net>>
>>>>
>>>> For additional commands, e-mail:
>>>> users-help_at_grizzly.dev.java.net
>>>> <mailto:users-help_at_grizzly.dev.java.net>
>>>> <mailto:users-help_at_grizzly.dev.java.net
>>>> <mailto:users-help_at_grizzly.dev.java.net>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> <mailto:users-unsubscribe_at_grizzly.dev.java.net>
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>> <mailto:users-help_at_grizzly.dev.java.net>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>> <
>>>> proto
>>>> .jar
>>>> >
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>
>>
>
>