Hi Shreedhar.
I think that my current patch which I attached recently through dev-alias maybe resolved it.
Solution is that though member receives ADD_EVENT, member checks the previous view like MASTER_CHANGE_EVENT.
Of course, there can be better solution. But in the current code, I think that this solution is the simplest choice without big changes.
But like MASTER_CHANGE_EVENT, there can be additional load. e.g) for loop, String.contains(), etc...
But because ADD_EVENT, MASTER_CHANGE_EVENT and JOINED_AND_READY_EVENT is occurred rarely, So it seems not bad.
Thanks
--
Bongjae Chang
----- Original Message -----
From: Shreedhar Ganapathy
To: users_at_shoal.dev.java.net
Sent: Monday, June 30, 2008 3:28 PM
Subject: Re: [Shoal-Users] Still not sure it's working
Hi Bongjae
You are right about this fix being partial and I saw this earlier today while running a simple test.
We do need to think about this a bit more in terms of duplicate joins.
I will look into the provider code tomorrow in MasterNode to see how it treats members who are not in the previous view but for whom an Add event has not been fired.
If you get to it sooner, let me know.
Thanks
Shreedhar
Bongjae Chang wrote:
Hi Shreedhar and Mike.
I know Shreedhar is waiting for Mike's response about Issue #66(
https://shoal.dev.java.net/issues/show_bug.cgi?id=66)
But I think maybe this patch is not enough.
I expect that Server2 will receive duplicated join notification.
Here is my logs.
Assume that Server1's server token is 517a0864-1a11-4474-b40a-aff1ce79c12e and Server2's server token is 347f676f-5300-4f40-a409-7ee2d7882c14.
[Server1]
-------------------------------------------------------------------
2008. 6. 30 오후 1:22:57 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Starting SimpleJoinTest....
2008. 6. 30 오후 1:22:58 com.sun.enterprise.shoal.jointest.SimpleJoinTest initializeGMS
정보: Initializing Shoal for member: 517a0864-1a11-4474-b40a-aff1ce79c12e group:TestGroup
2008. 6. 30 오후 1:22:58 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Registering for group event notifications
2008. 6. 30 오후 1:22:58 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Joining Group TestGroup
2008. 6. 30 오후 1:22:59 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
정보: GMS View Change Received for group TestGroup : Members in view for (before change analysis) are :
1: MemberId: 517a0864-1a11-4474-b40a-aff1ce79c12e, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEAE124E2EAD144600A358C3E2A889E9F203
2008. 6. 30 오후 1:22:59 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
정보: Analyzing new membership snapshot received as part of event : MASTER_CHANGE_EVENT
***JoinNotification received: GroupLeader = false, Signal.getMemberToken() = 517a0864-1a11-4474-b40a-aff1ce79c12e, ServerName = 517a0864-1a11-4474-b40a-aff1ce79c12e, Leader = 517a0864-1a11-4474-b40a-aff1ce79c12e
2008. 6. 30 오후 1:23:05 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
정보: GMS View Change Received for group TestGroup : Members in view for (before change analysis) are :
1: MemberId: 517a0864-1a11-4474-b40a-aff1ce79c12e, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEAE124E2EAD144600A358C3E2A889E9F203
2: MemberId: 347f676f-5300-4f40-a409-7ee2d7882c14, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEC059DA8E0EC34A5696DB9C670EAB932503
2008. 6. 30 오후 1:23:06 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
정보: Analyzing new membership snapshot received as part of event : ADD_EVENT
***JoinNotification received: GroupLeader = true, Signal.getMemberToken() = 347f676f-5300-4f40-a409-7ee2d7882c14, ServerName = 517a0864-1a11-4474-b40a-aff1ce79c12e, Leader = 517a0864-1a11-4474-b40a-aff1ce79c12e
-------------------------------------------------------------------
[Server2]
-------------------------------------------------------------------
2008. 6. 30 오후 1:23:04 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Starting SimpleJoinTest....
2008. 6. 30 오후 1:23:05 com.sun.enterprise.shoal.jointest.SimpleJoinTest initializeGMS
정보: Initializing Shoal for member: 347f676f-5300-4f40-a409-7ee2d7882c14 group:TestGroup
2008. 6. 30 오후 1:23:05 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Registering for group event notifications
2008. 6. 30 오후 1:23:05 com.sun.enterprise.shoal.jointest.SimpleJoinTest runSimpleSample
정보: Joining Group TestGroup
2008. 6. 30 오후 1:23:05 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
정보: GMS View Change Received for group TestGroup : Members in view for (before change analysis) are :
1: MemberId: 347f676f-5300-4f40-a409-7ee2d7882c14, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEC059DA8E0EC34A5696DB9C670EAB932503
2008. 6. 30 오후 1:23:05 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
정보: Analyzing new membership snapshot received as part of event : MASTER_CHANGE_EVENT
***JoinNotification received: GroupLeader = false, Signal.getMemberToken() = 347f676f-5300-4f40-a409-7ee2d7882c14, ServerName = 347f676f-5300-4f40-a409-7ee2d7882c14, Leader = 347f676f-5300-4f40-a409-7ee2d7882c14
2008. 6. 30 오후 1:23:06 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
정보: GMS View Change Received for group TestGroup : Members in view for (before change analysis) are :
1: MemberId: 517a0864-1a11-4474-b40a-aff1ce79c12e, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEAE124E2EAD144600A358C3E2A889E9F203
2: MemberId: 347f676f-5300-4f40-a409-7ee2d7882c14, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEC059DA8E0EC34A5696DB9C670EAB932503
2008. 6. 30 오후 1:23:06 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
정보: Analyzing new membership snapshot received as part of event : MASTER_CHANGE_EVENT
***JoinNotification received: GroupLeader = false, Signal.getMemberToken() = 517a0864-1a11-4474-b40a-aff1ce79c12e, ServerName = 347f676f-5300-4f40-a409-7ee2d7882c14, Leader = 517a0864-1a11-4474-b40a-aff1ce79c12e
2008. 6. 30 오후 1:23:06 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
정보: GMS View Change Received for group TestGroup : Members in view for (before change analysis) are :
1: MemberId: 517a0864-1a11-4474-b40a-aff1ce79c12e, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEAE124E2EAD144600A358C3E2A889E9F203
2: MemberId: 347f676f-5300-4f40-a409-7ee2d7882c14, MemberType: CORE, Address: urn:jxta:uuid-0836778E36C54F728D5B934A965395CEC059DA8E0EC34A5696DB9C670EAB932503
2008. 6. 30 오후 1:23:06 com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
정보: Analyzing new membership snapshot received as part of event : ADD_EVENT
***JoinNotification received: GroupLeader = false, Signal.getMemberToken() = 347f676f-5300-4f40-a409-7ee2d7882c14, ServerName = 347f676f-5300-4f40-a409-7ee2d7882c14, Leader = 517a0864-1a11-4474-b40a-aff1ce79c12e
-------------------------------------------------------------------
The problem is the following.
Sever1 and Server2 will receive own join notifications by current changes which Shreedhar edited.
But after they receive own join notifications from first MASTER_CHANGE_EVENT , Server1 will send Server2 ADD_EVENT because Server1 is the master and is aware of Server2 newly.
Then Server2 will receive ADD_EVENT from Server1 which is the master and notify duplicate join notification.
So we can know that though any member receives ADD_EVENT from master, the member doesn't have to notify join notification without checking view history's changes. The member should always notify join notification by view histroy's changes like MASTER_CHANGE_EVENT.
Or Shreedhar's current patch should be applied to only master.
I expect that maybe Mike will obtain similar result. :-)
Thanks.
--
Bongjae Chang
----- Original Message -----
From: Shreedhar Ganapathy
To: users_at_shoal.dev.java.net
Sent: Saturday, June 28, 2008 6:42 AM
Subject: Re: [Shoal-Users] Still not sure it's working
Hi Mike
I have checked in a fix to ViewWindow where some wierd logic earlier was excluding the first view change from being notified.
Could you try with the latest sources?
Thanks
Shreedhar
Shreedhar Ganapathy wrote:
Thanks for bringing this up. We may have a bug in our internal test harness which should be reporting these issues and hence we have not seen it yet. Let me check.
Could you meantime file an issue in the tracker?
I know Bongjae Chang had made some observations on this a while ago and his patch is awaiting review and integration. That might have a fix.
Thanks
Shreedhar
Mike Wannamaker wrote:
I believe it’s a bug then. In both accounts. My SERVER-1 only gets a SERVER-1 message and SERVER-2 only gets a SERVER-1 message.
So when I say it only gets a SERVER-1 message I mean that the method
public void processNotification(Signal p_Signal)
is only being called with that message.
So if I start just SERVER-1, I see the GMS View Changed message, with just the SERVER-1 in it, but my processNotification(…) is not called. Not until I start SERVER-2 does it get called.
On SERVER-2, I see the original GMS View Changed with just SERVER-2, and then GMS View Changed with SERVER-1,SERVER-2, but only get one processNotification(…) call.
I will investigate further next week, but if you could have a look that would be great. Is no one else seeing this?
This is my processNotification() method
public void processNotification(Signal p_Signal)
{
try
{
p_Signal.acquire();
SignalLogger log = new SignalLogger(p_Signal);
log.logIt();
if(p_Signal instanceof MessageSignal)
{
MessageSignal msgSig = (MessageSignal)p_Signal;
String sMember = msgSig.getMemberToken();
Object o = ObjectUtil.toObject(msgSig.getMessage());
if(o instanceof SMessage)
{
SMessage smsg = (SMessage)o;
InetAddress sender = m_hmMembers.get(sMember).address;
smsg.setSender(sender);
SMessageLogger.log.systemInfo(getClass(), "FireMessage: " + smsg);
fireMessageReceived(smsg);
//fireMessageReceived(smsg);
}
else
{
SMessageLogger.log.systemInfo(getClass(), "Message is NOT SMessage??");
}
}
else if(p_Signal instanceof JoinNotificationSignal)
{
JoinNotificationSignal joinSig = (JoinNotificationSignal)p_Signal;
processClusterNotification();
}
else if(p_Signal instanceof JoinedAndReadyNotificationSignal)
{
JoinedAndReadyNotificationSignal joinSig = (JoinedAndReadyNotificationSignal)p_Signal;
processClusterNotification();
}
else if(p_Signal instanceof FailureSuspectedSignal)
{
FailureSuspectedSignal suspectSig = (FailureSuspectedSignal)p_Signal;
processClusterNotification();
}
else if(p_Signal instanceof FailureRecoverySignal)
{
FailureRecoverySignal failureSig = (FailureRecoverySignal)p_Signal;
processClusterNotification();
}
else if(p_Signal instanceof FailureNotificationSignal)
{
FailureNotificationSignal failureSig = (FailureNotificationSignal)p_Signal;
processClusterNotification();
}
else if(p_Signal instanceof PlannedShutdownSignal)
{
PlannedShutdownSignal shutdownSig = (PlannedShutdownSignal)p_Signal;
processClusterNotification();
}
else
{
SMessageLogger.log.debug(getClass(), "Received Notification of type : " + p_Signal.getClass().getName() + " Server: " + p_Signal.getMemberToken());
}
}
catch(SignalAcquireException e)
{
SMessageLogger.log.fatal(getClass(), "Exception occured while acquiring signal", e);
}
finally
{
try
{
p_Signal.release();
}
catch(SignalReleaseException e)
{
SMessageLogger.log.warn(getClass(), "Exception occured while releasing signal" , e);
}
}
}
----------------------------------------------------------------------
From: Shreedhar.Ganapathy_at_Sun.COM [mailto:Shreedhar.Ganapathy_at_Sun.COM]
Sent: June 27, 2008 11:21 AM
To: users_at_shoal.dev.java.net
Subject: Re: [Shoal-Users] Still not sure it's working
Hi Mike
The expected behavior is that as each server starts, its registered GMS client components will be notified of the server's own joining the group and any subsequent joins of other members.
So in essence, server-1 GMS clients should see a JoinNotificationSignal for server-1, and another for server-2
and in server-2, GMS clients should see a JoinNotificationSignal for server-2 and another for server-1.
The order here does not matter but correctness is important and if not its a bug to be fixed.
In the log below, Server-1 seems to be getting its own JoinNotificationSignal which is correct. Does it ever get the JoinNotificationSignal for server-2?
On server-2, I am seeing correct behavior.
(Ignore the log statements that show the view contents, as that is an event coming from the provider implementation - GMS notification signals are the ones that GMS clients should look in for correctness).
Let me know.
Thanks
Shreedhar
Mike Wannamaker wrote:
Hi Guys,
I’m still not sure it’s working as it’s supposed to? But maybe it is?
Start SERVER-1
Start SERVER-2
On SERVER-1 I get a JoinMessage but it is from SERVER-1?
On SERVER-2 I get a Join Message from SERVER-1, which is what I would expect?
Is this correct? This depends on when the two servers are started. If I wait for a period between startups I get SERVER-2 startup message on SERVER-1 and SERVER-1 startup message on SERVER-2. But if I start them both at the same time I get the above behaviour?
Starting both at virtually the same time I get …
SEVER-1 Output:
27-Jun-2008 12:40:29 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
INFO: GMS View Change Received for group RCS_CLUSTER : Members in view for (before change analysis) are :
1: MemberId: SERVER-1, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FDB946A28335F0413BBF73B77CCC8BFEC603
27-Jun-2008 12:40:29 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
INFO: Analyzing new membership snapshot received as part of event : MASTER_CHANGE_EVENT
27-Jun-2008 12:40:38 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
INFO: GMS View Change Received for group RCS_CLUSTER : Members in view for (before change analysis) are :
1: MemberId: SERVER-2, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FD0D4B867250FF460C9B539A161779845B03
2: MemberId: SERVER-1, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FDB946A28335F0413BBF73B77CCC8BFEC603
27-Jun-2008 12:40:38 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
INFO: Analyzing new membership snapshot received as part of event : ADD_EVENT
27-Jun-2008 12:40:38 AM DEBUG [pool-1-thread-1] com.opentext.ecm.services.smessage.impl.shoal.SignalLogger - - SERVER-1 >> JoinNotificationSignalImpl @ 27/06/08 12:40 AM - [RCS_CLUSTER]: (Hashtable:[(String:server.name)<-->(String:SERVER-1), (String:local.host)<-->(Inet4Address:mwana0061/10.6.2.89)])
Server-2 Output
27-Jun-2008 12:40:30 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
INFO: GMS View Change Received for group RCS_CLUSTER : Members in view for (before change analysis) are :
1: MemberId: SERVER-2, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FD0D4B867250FF460C9B539A161779845B03
27-Jun-2008 12:40:30 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
INFO: Analyzing new membership snapshot received as part of event : MASTER_CHANGE_EVENT
27-Jun-2008 12:40:38 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow getMemberTokens
INFO: GMS View Change Received for group RCS_CLUSTER : Members in view for (before change analysis) are :
1: MemberId: SERVER-2, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FD0D4B867250FF460C9B539A161779845B03
2: MemberId: SERVER-1, MemberType: CORE, Address: urn:jxta:uuid-2F39FF376B6A43E3905DAFC81B7D02FDB946A28335F0413BBF73B77CCC8BFEC603
27-Jun-2008 12:40:38 AM com.sun.enterprise.ee.cms.impl.jxta.ViewWindow newViewObserved
INFO: Analyzing new membership snapshot received as part of event : ADD_EVENT
27-Jun-2008 12:40:44 AM DEBUG [pool-1-thread-1] com.opentext.ecm.services.smessage.impl.shoal.SignalLogger - - SERVER-1 >> JoinNotificationSignalImpl @ 27/06/08 12:40 AM - [RCS_CLUSTER]: (Hashtable:[(String:server.name)<-->(String:SERVER-1), (String:local.host)<-->(Inet4Address:mwana0061/10.6.2.89)])