Hi Simon,
Simon Trudeau wrote:
> After investigating further, I guess you are right, the server seems to
> terminate before the connect method has a chance to complete. It is very
> interesting because it only happens on machine with 2 or more cores... I
> can't reproduce it on my single core machine.
>
> My best bet so far is that it is related to my CountDownLatch used to
> control my test termination and the shutdown of the Controller. From my
> point of view, it looks like the client selectorHandler's FilterChain
> gets ran twice for each messages sent to the server. That would explain
> why the application always crashes when I almost reaches half my number
> of connections.
Could it be the continuousExecution = true that produce the twice
invocation? I suspect that's the problem.
>
> My guess is that since both client and servers selectorHandlers are
> registered with the same Controller on the same machine, and since the
> client selectorHandler is a TCPSelectorHandler and is not binded to a
> specific port like for the server TCPSelectorHandler, than it must get
> invoked when the server receives a message and when the client receives
> a message! Does that make sense?
I don't think that can happen, as its the TCPSelectorHandler that accept
the connection and own the SelectionKey. One way to find it is to do a
System.out inside your Filter of Context.getSelectorHandler() to see if
this is the right SelectorHandler that invoke your Filter.
>
> So, if all my previous assumptions are right, how do I bind a
> selectorHandler on the client side to a specific set of ports (only the
> ones used by my clients to receive responses)? How can that be done if
> the receiving port only gets dynamically assigned once the client
> connection has been established?
I don't think this is the problem :-)
>
> From our previous discussions, if possible and recommended, I would like
> to share one stateless selectorHandler for all clients (see previously
> sent source code).
>
> Would using one controller for all clients and a different controller
> for the server changes anything?
No.
>
> What do you think?
I need more information :-) Can you check which TCPSelectorHandler is
invoking your Filter? Are the ProtocolChain shared between your
TCPSelectorHandler or you have two instances, one for each?
Thanks!
-- jeanfrancois
>
>
>
> Simon
>
> -----Original Message-----
> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
> Sent: March-07-08 2:37 PM
> To: users_at_grizzly.dev.java.net
> Subject: Re: Performance test failing due to race condition problems
>
> Salut,
>
> Simon Trudeau wrote:
>> I am trying to concurrently connect to 2500 servers at a time using my
>
>> client application. Unfortunately, I run into all sorts of instability
>
>> issues:
>>
>
> On which platform are you. Those clients all try to connect to remove
> server, right (no local server)?
>
>>
>>
>> After connecting my 1268th client (it may vary) to the server, I get
> the
>> following exception:
>>
>>
>>
>> Exception in thread "pool-1-thread-10"
>> java.nio.channels.NotYetConnectedException
>>
>> at
>>
> com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:387)
>> ...
>>
>> java.nio.channels.ClosedChannelException
>>
>> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown
> Source)
>> at
>>
> com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
> va:565)
>> at
>>
> client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
> .java:214)
>>
>>
>> Running the same test I obtain also:
>
> It seems your server close the connection before you have a chance to
> finish the connect method.
>
>
>
>>
>>
>> After connecting my 1253th client (it may vary) to the server, I get
> the
>> following exception:
>>
>>
>>
>> java.nio.channels.ClosedChannelException
>>
>> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown
> Source)
>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>>
>> at
>>
> com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:403)
>> ...
>>
>> 7-Mar-2008 1:48:04 PM com.sun.grizzly.TCPConnectorHandler
> configureChannel
>> WARNING: setTcpNoDelay exception
>>
>> java.net.SocketException: Connection reset by peer:
>> sun.nio.ch.Net.setIntOption
>>
>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>
>> at sun.nio.ch.Net.setIntOption(Unknown Source)
>>
>> at sun.nio.ch.SocketChannelImpl$1.setInt(Unknown Source)
>>
>> at sun.nio.ch.SocketOptsImpl.setBoolean(Unknown Source)
>>
>> at sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(Unknown
> Source)
>> at sun.nio.ch.OptionAdaptor.setTcpNoDelay(Unknown Source)
>>
>> at sun.nio.ch.SocketAdaptor.setTcpNoDelay(Unknown Source)
>>
>> at
>>
> com.sun.grizzly.TCPConnectorHandler.configureChannel(TCPConnectorHandler
> .java:596)
>> at
>>
> com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
> va:567)
>> at
>>
> client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
> .java:214)
>> at
>>
> com.sun.grizzly.CallbackHandlerContextTask.doCall(CallbackHandlerContext
> Task.java:66)
>> at
>>
> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.jav
> a:57)
>> at
>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:179)
>
>
> That one should not be the issue. How many file descriptor you machine
> enable you right now:
>
> % unlimit -l
>
>>
>>
>> I also get for the same test
>>
>>
>>
>> After connecting my 1253th client (it may vary) to the server, I get
> the
>> following exception:
>>
>>
>>
>> Exception in thread "pool-1-thread-5" java.lang.IllegalStateException:
>
>> SelectorHandler not yet started
>>
>> at
>>
> com.sun.grizzly.TCPSelectorHandler.acquireConnectorHandler(TCPSelectorHa
> ndler.java:778)
>> at
>> client.BtNIOClient$Connector.initConnector(BtNIOClient.java:181)
>>
>>
>>
>> What's weird is that it always happens around my 1250-1260 th
> client...
>>
>>
>> I am testing with both client and server using the same Controller.
>> Tests are ran on a dual core intel machine. You need to run the
>> performance test (ClientPerformanceTest.java performanceTest1()) a few
>
>> times to get the exceptions, they don't occur on each run... which is
>> weird but which is also consistent with the race condition problems I
> am
>> encountering.
>>
>>
>>
>> I have attached my full source code with test to this mail so you can
>> run my test and maybe some of you might help me figure out what I did
> wrong.
>>
>>
>> To run the test, just include on the classpath the latest
>> grizzly-framework and use java 6.
>
> OK will try to take a look today....
>
> -- Jeanfrancois
>
>
>
>
>>
>>
>>
>>
>> Thanks,
>>
>>
>>
>>
>>
>> Simon
>>
>>
>>
> ------------------------------------------------------------------------
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>