Simon Trudeau wrote:
> I will investigate that issue today. I will get back to you on your
> question by the end of today. Should I want to show you source code and
> unit tests, what it the best way to do so? Would an eclipse project do
> the trick or would you prefer a Maven project (I am setup for both so
> that shouldn't be a problem)?
Maven :-) I'm still a vim user :-)
A+
-- jeanfrancois
>
> Simon
>
> -----Original Message-----
> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
> Sent: March-10-08 8:56 PM
> To: users_at_grizzly.dev.java.net
> Subject: Re: Performance test failing due to race condition problems
>
> Hi Simon,
>
> Simon Trudeau wrote:
>> After investigating further, I guess you are right, the server seems
> to
>> terminate before the connect method has a chance to complete. It is
> very
>> interesting because it only happens on machine with 2 or more cores...
> I
>> can't reproduce it on my single core machine.
>>
>> My best bet so far is that it is related to my CountDownLatch used to
>> control my test termination and the shutdown of the Controller. From
> my
>> point of view, it looks like the client selectorHandler's FilterChain
>> gets ran twice for each messages sent to the server. That would
> explain
>> why the application always crashes when I almost reaches half my
> number
>> of connections.
>
> Could it be the continuousExecution = true that produce the twice
> invocation? I suspect that's the problem.
>
>
>> My guess is that since both client and servers selectorHandlers are
>> registered with the same Controller on the same machine, and since the
>> client selectorHandler is a TCPSelectorHandler and is not binded to a
>> specific port like for the server TCPSelectorHandler, than it must get
>> invoked when the server receives a message and when the client
> receives
>> a message! Does that make sense?
>
> I don't think that can happen, as its the TCPSelectorHandler that accept
>
> the connection and own the SelectionKey. One way to find it is to do a
> System.out inside your Filter of Context.getSelectorHandler() to see if
> this is the right SelectorHandler that invoke your Filter.
>
>
>> So, if all my previous assumptions are right, how do I bind a
>> selectorHandler on the client side to a specific set of ports (only
> the
>> ones used by my clients to receive responses)? How can that be done if
>> the receiving port only gets dynamically assigned once the client
>> connection has been established?
>
> I don't think this is the problem :-)
>
>
>> From our previous discussions, if possible and recommended, I would
> like
>> to share one stateless selectorHandler for all clients (see previously
>> sent source code).
>>
>> Would using one controller for all clients and a different controller
>> for the server changes anything?
>
> No.
>
>> What do you think?
>
> I need more information :-) Can you check which TCPSelectorHandler is
> invoking your Filter? Are the ProtocolChain shared between your
> TCPSelectorHandler or you have two instances, one for each?
>
> Thanks!
>
> -- jeanfrancois
>
>
>>
>>
>> Simon
>>
>> -----Original Message-----
>> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>
>> Sent: March-07-08 2:37 PM
>> To: users_at_grizzly.dev.java.net
>> Subject: Re: Performance test failing due to race condition problems
>>
>> Salut,
>>
>> Simon Trudeau wrote:
>>> I am trying to concurrently connect to 2500 servers at a time using
> my
>>> client application. Unfortunately, I run into all sorts of
> instability
>>> issues:
>>>
>> On which platform are you. Those clients all try to connect to remove
>> server, right (no local server)?
>>
>>>
>>>
>>> After connecting my 1268th client (it may vary) to the server, I get
>> the
>>> following exception:
>>>
>>>
>>>
>>> Exception in thread "pool-1-thread-10"
>>> java.nio.channels.NotYetConnectedException
>>>
>>> at
>>>
> com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:387)
>>> ...
>>>
>>> java.nio.channels.ClosedChannelException
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown
>> Source)
>>> at
>>>
> com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
>> va:565)
>>> at
>>>
> client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
>> .java:214)
>>>
>>>
>>> Running the same test I obtain also:
>> It seems your server close the connection before you have a chance to
>> finish the connect method.
>>
>>
>>
>>>
>>>
>>> After connecting my 1253th client (it may vary) to the server, I get
>> the
>>> following exception:
>>>
>>>
>>>
>>> java.nio.channels.ClosedChannelException
>>>
>>> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown
>> Source)
>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>>>
>>> at
>>>
> com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:403)
>>> ...
>>>
>>> 7-Mar-2008 1:48:04 PM com.sun.grizzly.TCPConnectorHandler
>> configureChannel
>>> WARNING: setTcpNoDelay exception
>>>
>>> java.net.SocketException: Connection reset by peer:
>>> sun.nio.ch.Net.setIntOption
>>>
>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>
>>> at sun.nio.ch.Net.setIntOption(Unknown Source)
>>>
>>> at sun.nio.ch.SocketChannelImpl$1.setInt(Unknown Source)
>>>
>>> at sun.nio.ch.SocketOptsImpl.setBoolean(Unknown Source)
>>>
>>> at sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(Unknown
>> Source)
>>> at sun.nio.ch.OptionAdaptor.setTcpNoDelay(Unknown Source)
>>>
>>> at sun.nio.ch.SocketAdaptor.setTcpNoDelay(Unknown Source)
>>>
>>> at
>>>
> com.sun.grizzly.TCPConnectorHandler.configureChannel(TCPConnectorHandler
>> .java:596)
>>> at
>>>
> com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
>> va:567)
>>> at
>>>
> client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
>> .java:214)
>>> at
>>>
> com.sun.grizzly.CallbackHandlerContextTask.doCall(CallbackHandlerContext
>> Task.java:66)
>>> at
>>>
> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.jav
>> a:57)
>>> at
>>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:179)
>>
>> That one should not be the issue. How many file descriptor you machine
>
>> enable you right now:
>>
>> % unlimit -l
>>
>>>
>>>
>>> I also get for the same test
>>>
>>>
>>>
>>> After connecting my 1253th client (it may vary) to the server, I get
>> the
>>> following exception:
>>>
>>>
>>>
>>> Exception in thread "pool-1-thread-5"
> java.lang.IllegalStateException:
>>> SelectorHandler not yet started
>>>
>>> at
>>>
> com.sun.grizzly.TCPSelectorHandler.acquireConnectorHandler(TCPSelectorHa
>> ndler.java:778)
>>> at
>>> client.BtNIOClient$Connector.initConnector(BtNIOClient.java:181)
>>>
>>>
>>>
>>> What's weird is that it always happens around my 1250-1260 th
>> client...
>>>
>>>
>>> I am testing with both client and server using the same Controller.
>>> Tests are ran on a dual core intel machine. You need to run the
>>> performance test (ClientPerformanceTest.java performanceTest1()) a
> few
>>> times to get the exceptions, they don't occur on each run... which is
>
>>> weird but which is also consistent with the race condition problems I
>> am
>>> encountering.
>>>
>>>
>>>
>>> I have attached my full source code with test to this mail so you can
>
>>> run my test and maybe some of you might help me figure out what I did
>> wrong.
>>>
>>>
>>> To run the test, just include on the classpath the latest
>>> grizzly-framework and use java 6.
>> OK will try to take a look today....
>>
>> -- Jeanfrancois
>>
>>
>>
>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>>
>>>
>>> Simon
>>>
>>>
>>>
> ------------------------------------------------------------------------
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>