Jerry Raj wrote:
>
>
<deleted previous threads>
> Right, members can come and go as they please. I simply need a deterministic way
> that a new member can use to broadcast a piece of info to existing members, and
> be sure that all existing members will receive it. I can live with some
> uncertainty over members that joined at the same time or are in the process of
> leaving, but those that joined a while ago and are not leaving should receive
> the broadcast.
>
Jerry,
Thanks for composing this requirement. I will work on a solution for this.
-Joe
> -Jerry
>
>
>> All solutions that I had with synchronizing broadcasting messages in JMS
>> included knowledge in the sender
>> of how many other processes it was waiting for. The initial message
>> sent out was not a content message, but
>> a STARTUP message. When all the other processes replied back to the
>> STARTUP message, the application knew
>> that all loosely coupled processes all had their JMS subscriptions
>> initialized and were ready to proceed.
>>
>>
>> -Joe
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> Thanks
>>> -Jerry
>>>
>>>
>>>
>>>
>>>> -Joe Fialli
>>>>
>>>>
>>>>
>>>>> I have a little different opinion.
>>>>>
>>>>> I think that the discovery should also be finished for completing join.
>>>>> "Join" logic includes a lot of initializations which include
>>>>> MasterNode and ClusterViewManager's init.
>>>>>
>>>>> When gms.join() is called, MasterNode's init will be executed in
>>>>> separate thread and MasterNode's init starts ClusterViewManager.
>>>>>
>>>>> ClusterViewManager could be related to GroupHandler#sendMessage().
>>>>>
>>>>> What do you think, Joe?
>>>>>
>>>>> And Jerry,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>> Do you see a view change with the two members in it ?
>>>>>>>
>>>>>>>
>>>>>> Yes, I do.
>>>>>>
>>>>>>
>>>>> Though you could see a view change with the two members, if you send a
>>>>> message at that time, I think that the message could be lost. I am not
>>>>> sure.
>>>>>
>>>>> But maybe I think you set the shoal logger to be Level.FINE, the
>>>>> problem's reason could be found.
>>>>>
>>>>> And at first, you would like to check whether join is complete or not.
>>>>>
>>>>> Sometimes I used the following trick for this.
>>>>> GroupManagementService#reportJoinedAndReadyState() will be returned
>>>>> after the discovery time.
>>>>>
>>>>> So I am curious to know the following test's result.
>>>>>
>>>>> <snip>
>>>>> gms.join();
>>>>>
>>>>> gms.reportJoinedAndReadyState();
>>>>>
>>>>> // Commented: Thread.sleep(10000);
>>>>> GroupHandle gh = gms.getGroupHandle();
>>>>> gh.sendMessage(blah);
>>>>> </snip>
>>>>>
>>>>> Could you please test it again?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Bongjae Chang
>>>>>
>>>>>
>>>>> ----- Original Message ----- From: "Jerry Raj" <jerryr_at_sun.com>
>>>>> To: <users_at_shoal.dev.java.net>
>>>>> Sent: Friday, June 19, 2009 7:46 PM
>>>>> Subject: Re: [Shoal-Users] Re: send message after join
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Shreedhar Ganapathy wrote:
>>>>>>
>>>>>>
>>>>>>>>> Do you see a view change with the two members in it ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes, I do.
>>>>>>>>
>>>>>>>>
>>>>>>>>> If your send() is around the same time as the time the view change
>>>>>>>>> happens then you will have message loss.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Ah. This must be it. Since the join and send happen immediately one
>>>>>>>> after the
>>>>>>>> other, its quite likely that the send happens while the existing
>>>>>>>> peers are
>>>>>>>> processing the view change.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also, with Shoal 1.1 we have made a lot of synchronization
>>>>>>>>> improvements
>>>>>>>>> for correctness, which may have cost somewhat at the time of
>>>>>>>>> startup of
>>>>>>>>> members.
>>>>>>>>>
>>>>>>>>> Could you try out the code snippet that Joe provided as a way to
>>>>>>>>> ensure
>>>>>>>>> that message sending happens only when requisite memberships in
>>>>>>>>> the
>>>>>>>>> group are in place ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I have no pre-existing knowledge of the number of expected peers.
>>>>>>>> So waiting for
>>>>>>>> "all" of them to join has no real meaning for me. The idea here is
>>>>>>>> that peers
>>>>>>>> can join and leave as they please. The message they send as soon as
>>>>>>>> they join is
>>>>>>>> used by existing peers (if any) to gain certain data about the new
>>>>>>>> peer (its IP
>>>>>>>> address, port its listening on etc). So the code from below will
>>>>>>>> not really work
>>>>>>>> for me, since I have no idea if any more peers will join or not.
>>>>>>>>
>>>>>>>> I can live with having a sleep between join and send for now,
>>>>>>>> except it seems
>>>>>>>> rather non-deterministic: are we always sure that 5 secs is enough?
>>>>>>>> Under load
>>>>>>>> will this go up? A clear callback or signal that says "It is now
>>>>>>>> safe to send
>>>>>>>> messages" will be much better.
>>>>>>>>
>>>>>>>>
>>>>>>> Timing is always a challenge when there are distributed systems
>>>>>>> involved
>>>>>>> and larger the number of members, the more involved it gets to ensure
>>>>>>> virtual synchrony especially on asynchronous systems. We could
>>>>>>> eventually look at an ack based system for membership lifecycle
>>>>>>> events'
>>>>>>> view change messages, but there are costs with that as well when
>>>>>>> memberships are large.
>>>>>>>
>>>>>>> One suggestion that may make it a bit better is that you could use
>>>>>>> the
>>>>>>> joined and ready construct for letting the group know that you are
>>>>>>> ready
>>>>>>> to receive and send messages. i.e when a member has connected to the
>>>>>>> group you get a join notification signal. After this when the
>>>>>>> member is
>>>>>>> ready to send messages, it can call the
>>>>>>> reportJoinedAndReadyState() API
>>>>>>> from the GroupManagementService reference
>>>>>>> http://fisheye5.cenqua.com/browse/~raw,r=1.16/shoal/gms/src/java/com/sun/enterprise/ee/cms/core/GroupManagementService.java
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Members who have registered for receiving the joined and ready
>>>>>>> notification signal and have joined the group would get that
>>>>>>> notification and then can send out messages to that member.
>>>>>>>
>>>>>>> Hope this helps you get a bit closer to deterministic knowledge of
>>>>>>> when
>>>>>>> a member(or set of members is/are ready to receive messages.
>>>>>>>
>>>>>>>
>>>>>> This is the opposite of my problem. In my case, I have two nodes, and
>>>>>> the
>>>>>> following is the order of execution in chronological order:
>>>>>>
>>>>>> Node1:
>>>>>> t0: join()
>>>>>> t1: send() --> goes nowhere, as expected
>>>>>>
>>>>>> Node2:
>>>>>> t2: join()
>>>>>> t3: send() --> goes nowhere, not expected
>>>>>>
>>>>>> In this case Node1 can register for a joined and ready notification,
>>>>>> but it will
>>>>>> not help, since node1 is not going to send anything.
>>>>>>
>>>>>> -Jerry
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> -Jerry
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Shreedhar
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> The same identical code worked fine with Shoal 1.0.
>>>>>>>>>>
>>>>>>>>>> I hope this clarifies the use-case.
>>>>>>>>>> -Jerry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> However, there is not enough information in your original post to
>>>>>>>>>>> confirm or deny this.
>>>>>>>>>>>
>>>>>>>>>>> You could delay sending a message to the group until there is
>>>>>>>>>>> certain
>>>>>>>>>>> number of members joined or
>>>>>>>>>>> you could wait for all members to have joined via
>>>>>>>>>>> JoinNotification event.
>>>>>>>>>>>
>>>>>>>>>>> Pull API:
>>>>>>>>>>>
>>>>>>>>>>> List<String> members =
>>>>>>>>>>> gms.getGroupHandle().getAllCurrentMembers();
>>>>>>>>>>> <wait to send first message until all expected number of members
>>>>>>>>>>> have
>>>>>>>>>>> joined>
>>>>>>>>>>>
>>>>>>>>>>> Event driven API:
>>>>>>>>>>>
>>>>>>>>>>> gms.addActionFactory( new JoinNotificationActionFactoryImpl(
>>>>>>>>>>> new
>>>>>>>>>>> JoinNotificationCallBack( serverName ) ) );
>>>>>>>>>>> gms.join();
>>>>>>>>>>> <wait till all expected instances have joined before sending
>>>>>>>>>>> message;
>>>>>>>>>>> use info calculated from JoinNotificationCallback>
>>>>>>>>>>>
>>>>>>>>>>> private class JoinNotificationCallBack implements CallBack {
>>>>>>>>>>>
>>>>>>>>>>> private String serverName;
>>>>>>>>>>>
>>>>>>>>>>> public JoinNotificationCallBack( String serverName ) {
>>>>>>>>>>> this.serverName = serverName;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> // called for every instance joining the gms group.
>>>>>>>>>>> public void processNotification( Signal notification ) {
>>>>>>>>>>> <record instance has joined>;
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> A non-coding way to check this out is to start all your receiving
>>>>>>>>>>> clients first.
>>>>>>>>>>> Wait 10 seconds (like your initial test).
>>>>>>>>>>> Then start your sending gms client.
>>>>>>>>>>> There is no need a sleep between the join and send since all the
>>>>>>>>>>> other
>>>>>>>>>>> members
>>>>>>>>>>> will have already joined. Hope this helps.
>>>>>>>>>>>
>>>>>>>>>>> -Joe
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> -Jerry
>>>>>>>>>>>>
>>>>>>>>>>>> Jerry Raj wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I have code like this:
>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>> gms.join();
>>>>>>>>>>>>> // Commented: Thread.sleep(10000);
>>>>>>>>>>>>> GroupHandle gh = gms.getGroupHandle();
>>>>>>>>>>>>>
>>>>>>>>>>>>> gh.sendMessage(blah);
>>>>>>>>>>>>>
>>>>>>>>>>>>> </snip>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This used to work fine in Shoal 1.0. The node would join the
>>>>>>>>>>>>> group
>>>>>>>>>>>>> and the
>>>>>>>>>>>>> message would be recd by other members in the group. But this
>>>>>>>>>>>>> does
>>>>>>>>>>>>> not happen
>>>>>>>>>>>>> with Shoal 1.1 unless I uncomment the sleep(10000) between
>>>>>>>>>>>>> join() and
>>>>>>>>>>>>> send(). I
>>>>>>>>>>>>> expect this is because the join operation has not completed
>>>>>>>>>>>>> successfully when
>>>>>>>>>>>>> send() is called. Is there a way to be notified when join is
>>>>>>>>>>>>> complete? I tried
>>>>>>>>>>>>> looking at JoinedAndReadyNotificationActionImpl but that
>>>>>>>>>>>>> does not
>>>>>>>>>>>>> seem to work?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm using Shoal 1.1 from the download link on the front page
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> Shoal website.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jerry
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>>>>>> <mailto:users-unsubscribe_at_shoal.dev.java.net>
>>>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>>>>> <mailto:users-help_at_shoal.dev.java.net>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
>> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_shoal.dev.java.net
> For additional commands, e-mail: users-help_at_shoal.dev.java.net
>
>