After investigating further, I guess you are right, the server seems to
terminate before the connect method has a chance to complete. It is very
interesting because it only happens on machine with 2 or more cores... I
can't reproduce it on my single core machine.
My best bet so far is that it is related to my CountDownLatch used to
control my test termination and the shutdown of the Controller. From my
point of view, it looks like the client selectorHandler's FilterChain
gets ran twice for each messages sent to the server. That would explain
why the application always crashes when I almost reaches half my number
of connections.
My guess is that since both client and servers selectorHandlers are
registered with the same Controller on the same machine, and since the
client selectorHandler is a TCPSelectorHandler and is not binded to a
specific port like for the server TCPSelectorHandler, than it must get
invoked when the server receives a message and when the client receives
a message! Does that make sense?
So, if all my previous assumptions are right, how do I bind a
selectorHandler on the client side to a specific set of ports (only the
ones used by my clients to receive responses)? How can that be done if
the receiving port only gets dynamically assigned once the client
connection has been established?
From our previous discussions, if possible and recommended, I would like
to share one stateless selectorHandler for all clients (see previously
sent source code).
Would using one controller for all clients and a different controller
for the server changes anything?
What do you think?
Simon
-----Original Message-----
From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
Sent: March-07-08 2:37 PM
To: users_at_grizzly.dev.java.net
Subject: Re: Performance test failing due to race condition problems
Salut,
Simon Trudeau wrote:
> I am trying to concurrently connect to 2500 servers at a time using my
> client application. Unfortunately, I run into all sorts of instability
> issues:
>
On which platform are you. Those clients all try to connect to remove
server, right (no local server)?
>
>
> After connecting my 1268th client (it may vary) to the server, I get
the
> following exception:
>
>
>
> Exception in thread "pool-1-thread-10"
> java.nio.channels.NotYetConnectedException
>
> at
>
com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:387)
>
> ...
>
> java.nio.channels.ClosedChannelException
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown
Source)
>
> at
>
com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
va:565)
>
> at
>
client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
.java:214)
>
>
>
> Running the same test I obtain also:
It seems your server close the connection before you have a chance to
finish the connect method.
>
>
>
> After connecting my 1253th client (it may vary) to the server, I get
the
> following exception:
>
>
>
> java.nio.channels.ClosedChannelException
>
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown
Source)
>
> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>
> at
>
com.sun.grizzly.TCPConnectorHandler.write(TCPConnectorHandler.java:403)
>
> ...
>
> 7-Mar-2008 1:48:04 PM com.sun.grizzly.TCPConnectorHandler
configureChannel
>
> WARNING: setTcpNoDelay exception
>
> java.net.SocketException: Connection reset by peer:
> sun.nio.ch.Net.setIntOption
>
> at sun.nio.ch.Net.setIntOption0(Native Method)
>
> at sun.nio.ch.Net.setIntOption(Unknown Source)
>
> at sun.nio.ch.SocketChannelImpl$1.setInt(Unknown Source)
>
> at sun.nio.ch.SocketOptsImpl.setBoolean(Unknown Source)
>
> at sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(Unknown
Source)
>
> at sun.nio.ch.OptionAdaptor.setTcpNoDelay(Unknown Source)
>
> at sun.nio.ch.SocketAdaptor.setTcpNoDelay(Unknown Source)
>
> at
>
com.sun.grizzly.TCPConnectorHandler.configureChannel(TCPConnectorHandler
.java:596)
>
> at
>
com.sun.grizzly.TCPConnectorHandler.finishConnect(TCPConnectorHandler.ja
va:567)
>
> at
>
client.BtNIOClient$Connector$ClientCallBackHandler.onConnect(BtNIOClient
.java:214)
>
> at
>
com.sun.grizzly.CallbackHandlerContextTask.doCall(CallbackHandlerContext
Task.java:66)
>
> at
>
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.jav
a:57)
>
> at
> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:179)
That one should not be the issue. How many file descriptor you machine
enable you right now:
% unlimit -l
>
>
>
> I also get for the same test
>
>
>
> After connecting my 1253th client (it may vary) to the server, I get
the
> following exception:
>
>
>
> Exception in thread "pool-1-thread-5" java.lang.IllegalStateException:
> SelectorHandler not yet started
>
> at
>
com.sun.grizzly.TCPSelectorHandler.acquireConnectorHandler(TCPSelectorHa
ndler.java:778)
>
> at
> client.BtNIOClient$Connector.initConnector(BtNIOClient.java:181)
>
>
>
> What's weird is that it always happens around my 1250-1260 th
client...
>
>
>
> I am testing with both client and server using the same Controller.
> Tests are ran on a dual core intel machine. You need to run the
> performance test (ClientPerformanceTest.java performanceTest1()) a few
> times to get the exceptions, they don't occur on each run... which is
> weird but which is also consistent with the race condition problems I
am
> encountering.
>
>
>
> I have attached my full source code with test to this mail so you can
> run my test and maybe some of you might help me figure out what I did
wrong.
>
>
>
> To run the test, just include on the classpath the latest
> grizzly-framework and use java 6.
OK will try to take a look today....
-- Jeanfrancois
>
>
>
>
>
> Thanks,
>
>
>
>
>
> Simon
>
>
>
------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
For additional commands, e-mail: users-help_at_grizzly.dev.java.net