Hi,
Joe wrote:
>It is not that join is not complete for the sender, the sender is
>definitely part of the group when join returns.
I have a little different opinion.
I think that the discovery should also be finished for completing join.
"Join" logic includes a lot of initializations which include MasterNode and ClusterViewManager's init.
When gms.join() is called, MasterNode's init will be executed in separate thread and MasterNode's init starts ClusterViewManager.
ClusterViewManager could be related to GroupHandler#sendMessage().
What do you think, Joe?
And Jerry,
>> Do you see a view change with the two members in it ?
>Yes, I do.
Though you could see a view change with the two members, if you send a message at that time, I think that the message could be lost. I am not sure.
But maybe I think you set the shoal logger to be Level.FINE, the problem's reason could be found.
And at first, you would like to check whether join is complete or not.
Sometimes I used the following trick for this.
GroupManagementService#reportJoinedAndReadyState() will be returned after the discovery time.
So I am curious to know the following test's result.
<snip>
gms.join();
gms.reportJoinedAndReadyState();
// Commented: Thread.sleep(10000);
GroupHandle gh = gms.getGroupHandle();
gh.sendMessage(blah);
</snip>
Could you please test it again?
Thanks.
--
Bongjae Chang
----- Original Message -----
From: "Jerry Raj" <jerryr_at_sun.com>
To: <users_at_shoal.dev.java.net>
Sent: Friday, June 19, 2009 7:46 PM
Subject: Re: [Shoal-Users] Re: send message after join
>
>
> Shreedhar Ganapathy wrote:
>>
>>>> Do you see a view change with the two members in it ?
>>>>
>>> Yes, I do.
>>>
>>>> If your send() is around the same time as the time the view change
>>>> happens then you will have message loss.
>>>>
>>> Ah. This must be it. Since the join and send happen immediately one after the
>>> other, its quite likely that the send happens while the existing peers are
>>> processing the view change.
>>>
>>>
>>>> Also, with Shoal 1.1 we have made a lot of synchronization improvements
>>>> for correctness, which may have cost somewhat at the time of startup of
>>>> members.
>>>>
>>>> Could you try out the code snippet that Joe provided as a way to ensure
>>>> that message sending happens only when requisite memberships in the
>>>> group are in place ?
>>>>
>>> I have no pre-existing knowledge of the number of expected peers. So waiting for
>>> "all" of them to join has no real meaning for me. The idea here is that peers
>>> can join and leave as they please. The message they send as soon as they join is
>>> used by existing peers (if any) to gain certain data about the new peer (its IP
>>> address, port its listening on etc). So the code from below will not really work
>>> for me, since I have no idea if any more peers will join or not.
>>>
>>> I can live with having a sleep between join and send for now, except it seems
>>> rather non-deterministic: are we always sure that 5 secs is enough? Under load
>>> will this go up? A clear callback or signal that says "It is now safe to send
>>> messages" will be much better.
>>>
>> Timing is always a challenge when there are distributed systems involved
>> and larger the number of members, the more involved it gets to ensure
>> virtual synchrony especially on asynchronous systems. We could
>> eventually look at an ack based system for membership lifecycle events'
>> view change messages, but there are costs with that as well when
>> memberships are large.
>>
>> One suggestion that may make it a bit better is that you could use the
>> joined and ready construct for letting the group know that you are ready
>> to receive and send messages. i.e when a member has connected to the
>> group you get a join notification signal. After this when the member is
>> ready to send messages, it can call the reportJoinedAndReadyState() API
>> from the GroupManagementService reference
>> http://fisheye5.cenqua.com/browse/~raw,r=1.16/shoal/gms/src/java/com/sun/enterprise/ee/cms/core/GroupManagementService.java
>>
>> Members who have registered for receiving the joined and ready
>> notification signal and have joined the group would get that
>> notification and then can send out messages to that member.
>>
>> Hope this helps you get a bit closer to deterministic knowledge of when
>> a member(or set of members is/are ready to receive messages.
>
> This is the opposite of my problem. In my case, I have two nodes, and the
> following is the order of execution in chronological order:
>
> Node1:
> t0: join()
> t1: send() --> goes nowhere, as expected
>
> Node2:
> t2: join()
> t3: send() --> goes nowhere, not expected
>
> In this case Node1 can register for a joined and ready notification, but it will
> not help, since node1 is not going to send anything.
>
> -Jerry
>
>
>
>>> -Jerry
>>>
>>>
>>>> Thanks
>>>> Shreedhar
>>>>
>>>>
>>>>> The same identical code worked fine with Shoal 1.0.
>>>>>
>>>>> I hope this clarifies the use-case.
>>>>> -Jerry
>>>>>
>>>>>
>>>>>
>>>>>> However, there is not enough information in your original post to
>>>>>> confirm or deny this.
>>>>>>
>>>>>> You could delay sending a message to the group until there is certain
>>>>>> number of members joined or
>>>>>> you could wait for all members to have joined via JoinNotification event.
>>>>>>
>>>>>> Pull API:
>>>>>>
>>>>>> List<String> members = gms.getGroupHandle().getAllCurrentMembers();
>>>>>> <wait to send first message until all expected number of members have
>>>>>> joined>
>>>>>>
>>>>>> Event driven API:
>>>>>>
>>>>>> gms.addActionFactory( new JoinNotificationActionFactoryImpl( new
>>>>>> JoinNotificationCallBack( serverName ) ) );
>>>>>> gms.join();
>>>>>> <wait till all expected instances have joined before sending message;
>>>>>> use info calculated from JoinNotificationCallback>
>>>>>>
>>>>>> private class JoinNotificationCallBack implements CallBack {
>>>>>>
>>>>>> private String serverName;
>>>>>>
>>>>>> public JoinNotificationCallBack( String serverName ) {
>>>>>> this.serverName = serverName;
>>>>>> }
>>>>>>
>>>>>> // called for every instance joining the gms group.
>>>>>> public void processNotification( Signal notification ) {
>>>>>> <record instance has joined>;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> A non-coding way to check this out is to start all your receiving
>>>>>> clients first.
>>>>>> Wait 10 seconds (like your initial test).
>>>>>> Then start your sending gms client.
>>>>>> There is no need a sleep between the join and send since all the other
>>>>>> members
>>>>>> will have already joined. Hope this helps.
>>>>>>
>>>>>> -Joe
>>>>>>
>>>>>>
>>>>>>> Thanks
>>>>>>> -Jerry
>>>>>>>
>>>>>>> Jerry Raj wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hello,
>>>>>>>> I have code like this:
>>>>>>>> <snip>
>>>>>>>> gms.join();
>>>>>>>> // Commented: Thread.sleep(10000);
>>>>>>>> GroupHandle gh = gms.getGroupHandle();
>>>>>>>>
>>>>>>>> gh.sendMessage(blah);
>>>>>>>>
>>>>>>>> </snip>
>>>>>>>>
>>>>>>>> This used to work fine in Shoal 1.0. The node would join the group
>>>>>>>> and the
>>>>>>>> message would be recd by other members in the group. But this does
>>>>>>>> not happen
>>>>>>>> with Shoal 1.1 unless I uncomment the sleep(10000) between join() and
>>>>>>>> send(). I
>>>>>>>> expect this is because the join operation has not completed
>>>>>>>> successfully when
>>>>>>>> send() is called. Is there a way to be notified when join is
>>>>>>>> complete? I tried
>>>>>>>> looking at JoinedAndReadyNotificationActionImpl but that does not
>>>>>>>> seem to work?
>>>>>>>>
>>>>>>>> I'm using Shoal 1.1 from the download link on the front page of the
>>>>>>>> Shoal website.
>>>>>>>>
>>>>>>>> -Jerry
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net <mailto:users-unsubscribe_at_shoal.dev.java.net> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net <mailto:users-help_at_shoal.dev.java.net> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net <mailto:users-unsubscribe_at_shoal.dev.java.net> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net <mailto:users-help_at_shoal.dev.java.net> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net <mailto:users-unsubscribe_at_shoal.dev.java.net> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net <mailto:users-help_at_shoal.dev.java.net> <mailto:users-help_at_shoal.dev.java.net>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net <mailto:users-help_at_shoal.dev.java.net>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>
>
>
>