dev@grizzly.java.net

Re: Strange behavior of Grizzly

From: Bongjae Chang <carryel_at_korea.com>
Date: Thu, 11 Jun 2009 18:41:51 +0900

Hi Jeanfrancois and Minoru,

If I am understanding this issue correctly, I think that two issues exist now.

1) When the address which already has existed is bound again, sometimes the server can't receive packets any longer.
2) Though socket is bound successfully with different address, java.nio.channels.Selector.selectedKeys() returns empty set on windows XP in some circumestances.

Is it right?

First, I have looked into 1) case with testing my standalone program.

In my test, I found that it might be able to be not a JDK bug.

I used two Controllers and one client which was based on blocking IO.

At the result, the problem was caused by receiver's switch.

Here was my scenario.
the server Controller are receiving packets ->
the client Controller tries to connect and bind the same address -->
the server Controller doen't receive packets any longer -->
the client Controller are receiving packets instead of the server Controller.

When I tried to bind the same address with blocking DatagramSocket, I could meet the BindException.

When I tried to bind the same address with ConnectorHandler#connect( remote, local ), I could not meet the BindException because of setReuseAddress( true ).

If setReuseAddress is true(Grizzly's default is true), it seems that duplicated binding can be allowed.

Then sometimes the channel could be switched because the client's channel which was sending packets couldn't know where to send the packet to.
 
So I think that this is not JDK bug. What do you think?

I attached my stand-alone test program.

This program will reproduce 1) case simply and confirm that the receiver will be often switched.

Sometimes you can see the following result.

---
Wait for 2000(ms)
current thread: Thread[server-WorkerThread(0),5,Grizzly], Request: hello world-1
current thread: Thread[server-WorkerThread(5),5,Grizzly], Request: hello world-2
current thread: Thread[server-WorkerThread(6),5,Grizzly], Request: hello world-3
current thread: Thread[server-WorkerThread(7),5,Grizzly], Request: hello world-4
current thread: Thread[server-WorkerThread(8),5,Grizzly], Request: hello world-5
current thread: Thread[server-WorkerThread(9),5,Grizzly], Request: hello world-6
current thread: Thread[server-WorkerThread(10),5,Grizzly], Request: hello world-7
###Binding duplicated socket again..### --------------------------------------> receiver is switched ( server -> client )
current thread: Thread[client-WorkerThread(6),5,Grizzly], Request: hello world-8
current thread: Thread[client-WorkerThread(7),5,Grizzly], Request: hello world-9
current thread: Thread[client-WorkerThread(8),5,Grizzly], Request: hello world-10
current thread: Thread[client-WorkerThread(9),5,Grizzly], Request: hello world-11
current thread: Thread[client-WorkerThread(10),5,Grizzly], Request: hello world-12
---

And try to remove the code which has been commented out and comment out some codes for other test like the following.

In testSimpleSendAndReceive() method
---
// test #1
// reproduce the problem
// /*
for( int i=0; i<100; i++ ) {
   ...
}
// */

...

// test #3
// receiver's switch test
/*
ConnectorHandler connectorHandler = null;
...
*/
---

Second, I would like to have looked into 2) case, but unfortunately I couldn't reproduce it.

I hope that this test program helps you and I would like to know whether I am understanding this correctly or not.

Thanks.

--
Bongjae Chang


----- Original Message -----
From: "Jeanfrancois Arcand" <Jeanfrancois.Arcand_at_Sun.COM>
To: <dev_at_grizzly.dev.java.net>
Sent: Tuesday, June 09, 2009 12:52 AM
Subject: Re: Strange behavior of Grizzly


> Salut,
>
> Minoru Nitta wrote:
>> Hi Bongjae,
>>
>>> Hi, Minoru.
>>>
>>> I see..
>>>
>>> It's interesting. What is your JDK version?
>>
>>
>> I tested with 1.5 update 10 and 1.6 update 13.
>> I checked Sun Bug database but I could't find the issues, which might be
>> releavent to this.
>
> I've escalated the issue to the JDK team as I was also able to reproduce
> the issue.
>
> Bongjae, go ahead and commit your workaround. I will reply to that list
> once I have the JDK bug issue number or an explanation why it is failing.
>
> Good work!!
>
> -- Jeanfrancois
>
>
>>
>> Minoru
>>
>>> --
>>> Bongjae Chang
>>>
>>>
>>> ----- Original Message -----
>>> From: "Minoru Nitta" <minoru.nitta_at_jp.fujitsu.com>
>>> To: <dev_at_grizzly.dev.java.net>
>>> Sent: Monday, June 08, 2009 8:52 AM
>>> Subject: Re: Strange behavior of Grizzly
>>>
>>>
>>>> Hi Bongjae,
>>>>
>>>>
>>>>> Hi Bongjae,
>>>>>
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> I tested my original program
>>>>> on WindowsXP -> failed
>>>>> on Liunx -> succeeded
>>>>>
>>>>> After I applied your patch, my test program succeeded on both platforms.
>>>>> On Windows, my program failed because selector.select() returns empty set
>>>> Actually, java.nio.channels.Selector.selectedKeys().
>>>> Thanks.
>>>>
>>>>> on Windows. So in some circumestances, there might be a problem.
>>>>>
>>>>>
>>>>> Minoru
>>>>>
>>>>>> Hi Minoru,
>>>>>>
>>>>>> I tested your program on Windows 2003 server.
>>>>>>
>>>>>> Your original program failed, too. But when I applied my proposed patch to your program, I could know that it worked well.
>>>>>>
>>>>>> Didn't it work well when you tested your program which was modified by my opinion on Windows XP?
>>>>>>
>>>>>> --
>>>>>> Bongjae Chang
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Minoru Nitta" <minoru.nitta_at_jp.fujitsu.com>
>>>>>> To: <dev_at_grizzly.dev.java.net>
>>>>>> Sent: Friday, June 05, 2009 1:01 PM
>>>>>> Subject: Re: Strange behavior of Grizzly
>>>>>>
>>>>>>
>>>>>>> Hi Bongjae, Hi all,
>>>>>>>
>>>>>>>
>>>>>>> I ran my test program on RHEL4 and it worked fine. I ran it on Windows XP,
>>>>>>> it failed. I debugged it and found out that SelectorHandler.select returned
>>>>>>> empty set at SelectorHandlerRunner.doSelect even if ready keys have been
>>>>>>> registered (So, read operation can never be executed). I think this is a
>>>>>>> bug of JDK (or somewhere else).
>>>>>>>
>>>>>>> Did you kindly test my program on Windows?
>>>>>>>
>>>>>>> Minoru
>>>>>>>
>>>>>>>> Hi Bongjae,
>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you very much for your quick reply and curiosity. I understood
>>>>>>>> your proposal. Then, I would like to write my opinion.
>>>>>>>>
>>>>>>>> In UDP, it should be possible to send packets from a port where
>>>>>>>> server socket is already bound. For example, after starting my SIP server,
>>>>>>>> UDP ports status is like
>>>>>>>>
>>>>>>>> [root_at_rx200-4 ~]# netstat -a -u
>>>>>>>> Active Internet connections (servers and established)
>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>>>> udp 0 0 *:666 *:*
>>>>>>>> udp 0 0 *:669 *:*
>>>>>>>> udp 0 0 *:sunrpc *:*
>>>>>>>> udp 0 0 *:ipp *:*
>>>>>>>> udp 0 0 rx200-4:5060 *:*
>>>>>>>>
>>>>>>>> where
>>>>>>>>
>>>>>>>> udp 0 0 rx200-4:5060 *:*
>>>>>>>>
>>>>>>>> is a server socket where my SIP server is listening to from any other
>>>>>>>> addresses (*:*). Then after sending packets from the same port (i.e. 5060),
>>>>>>>> the status is like
>>>>>>>>
>>>>>>>> [root_at_rx200-4 ~]# netstat -a -u
>>>>>>>> Active Internet connections (servers and established)
>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>>>> udp 0 0 *:666 *:*
>>>>>>>> udp 0 0 *:669 *:*
>>>>>>>> udp 0 0 *:sunrpc *:*
>>>>>>>> udp 0 0 *:ipp *:*
>>>>>>>> udp 0 0 rx200-4:5060 [UNKNOWN]:6677 ESTABLISHED
>>>>>>>> udp 0 0 rx200-4:5060 *:*
>>>>>>>>
>>>>>>>> where
>>>>>>>>
>>>>>>>> udp 0 0 rx200-4:5060 [UNKNOWN]:6677 ESTABLISHED
>>>>>>>>
>>>>>>>> is a client socket which receives packets only from [UNKNOWN]:6677.
>>>>>>>>
>>>>>>>> So if packets are sent to the port 5060, the client socket is used
>>>>>>>> if the sender address is [UNKNOWN]:6677 and the server socket is used
>>>>>>>> if the sender address is not [UNKNOWN]:6677.
>>>>>>>>
>>>>>>>> I thought that the following code you mentioned in your reply makes
>>>>>>>> the
>>>>>>>>
>>>>>>>> udp 0 0 rx200-4:5060 [UNKNOWN]:6677 ESTABLISHED
>>>>>>>>
>>>>>>>> socket. But, I am not sure.
>>>>>>>>
>>>>>>>> Minoru
>>>>>>>>
>>>>>>>>> Hi Minoru,
>>>>>>>>>
>>>>>>>>> I saw your app curiously and I could reproduce your problem.
>>>>>>>>>
>>>>>>>>> When I debugged your program, I could know that your problem was perhaps caused by duplicated socket binding.
>>>>>>>>>
>>>>>>>>> I think that it is better that the client controller doesn't use a local address when it connects remote peer.
>>>>>>>>>
>>>>>>>>> See the following:
>>>>>>>>>
>>>>>>>>> In ConnectionManager.java
>>>>>>>>> ---
>>>>>>>>> private ConnectorHandler createHandlerUDP(...) {
>>>>>>>>> ...
>>>>>>>>> try {
>>>>>>>>> //connectorHandler.connect(socketRemote, local, callbackHandler);
>>>>>>>>> connectorHandler.connect(socketRemote, callbackHandler);
>>>>>>>>> } ...
>>>>>>>>> }
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> In my opinion, the server controller's socket which was already binded with a local address could have a problem if a client tried to bind own socket with the same address locally.
>>>>>>>>>
>>>>>>>>> I could see that your program's server couldn't receive any packets though blocking DatagramSocket(UAC and UAS) sent a packet to the server controller successfully.
>>>>>>>>>
>>>>>>>>> So I thought that maybe the server had a problem.
>>>>>>>>>
>>>>>>>>> It is not certain. :-)
>>>>>>>>>
>>>>>>>>> But, I hope that my proposed patch and my words help you.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bongjae Chang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Minoru Nitta" <minoru.nitta_at_jp.fujitsu.com>
>>>>>>>>> To: <dev_at_grizzly.dev.java.net>
>>>>>>>>> Sent: Tuesday, June 02, 2009 5:28 PM
>>>>>>>>> Subject: Strange behavior of Grizzly
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Salut,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have been encountering a strange behavior of grizzly, and I need help.
>>>>>>>>>> I tested the enclosed program and it sometimes worked fine and it sometimes did
>>>>>>>>>> not work well. When it works, it repeats sending requests and responses,
>>>>>>>>>> but when it does not work well, it stops repeating.
>>>>>>>>>>
>>>>>>>>>> I have no idea where is the problem (my test program or grizzly).
>>>>>>>>>> I tested it on grizzly version 1.9.12 to 1.9.16 and the result was same.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Minoru
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>
>
>
>