> Hi,
>
> In GFv2, EJB requests could be load balanced by passing multiple targets
> in the Hashtable parameter when instantiating InitialContext. (The
> targets would be randomized by rrPolicy.setClusterInstanceInfo).
Not true. There was never any supported feature to randomize or loadbalance
iiop targets in GFv2. The feature is FOLB (fail over load balancing) meaning
you connect to the first target past in always. Later if that fails, try the
next in the list, and so on until you get a new connection or fail.
>
> In GFv2.1 this does not happen anymore because of the following change
> by Mark in S1ASCtxFactory.java.
>
> 25184850
>
> Is there any particular reason for not doing the randomizing any more?
>
Yes there is.
The functionality you thought was IIOP load balancing did not work as you
thought, and was in fact FOLB (Fail Over Load Balancing) in which the server
should use FIFO logic to do initial connection and then (only in a fail over
situation) try the other connections passed in (IN FIFO order).
Out of this came the RFE request to have true load balancing of iiop
connections, a feature that does not exist yet.
See RFE 6794941 for IIOP LB and failover under GF cluster
"Engineering has agreed to implement another version of load balancing,
which we will refer to as per-request load balancing for stateless session beans."
Here is the technical explanation from the original bug about FOLB not working
and the customer's (and your) misconception that GFv2 ever supported
iiop lb.
Work Around
Can you go to the customer and suggest that they change their application code
to present the iiop.endpoints to the server in the order *they* want to connect to?
If they want to randomly connect to host1:iiop and to host2:iiop, why don't
they make their code do that? The system is designed to look at the
iiop connection string in FIFO order, and fail over ONLY on a connection error.
If they want to change the order to "load balance" they can do that by just
changing the order of the iiop connection endpoints they are supplying to the
server when they try to connect.
If they are creating a connection and caching it (to avoid the expensive
overhead of creating and tearing down the iiop endpoint connection each time)
then they don't even want to connect to different iiop hosts except during a
failover situation!
If they are creating and tearing down the connection each time, then just
randomizing the iiop.endpoints property should do what they want. See
the below example made from their sample code.
C:\temp>c:\bin\diff -u sample_code.txt lb_sample_code.txt
--- sample_code.txt Thu Jan 29 12:29:38 2009
+++ lb_sample_code.txt Thu Jan 29 12:29:28 2009
@@ -1,10 +1,19 @@
public static void main(String[] args) throws ServletException, IOException {
+ boolean coinflip = false; // variable to make client "load balance"
for (int i = 0; i < 100; i++) {
try {
Properties props = new Properties();
- props.put("com.sun.appserv.iiop.endpoints",
+ // change the order of iiop.endpoints based on the flip of a coin
+ if (coinflip) {
+ props.put("com.sun.appserv.iiop.endpoints",
"192.168.0.212:33700,192.168.0.218:33700"); // 212 failover to 218
+ coinflip = false;
+ } else {
+ props.put("com.sun.appserv.iiop.endpoints",
+ "192.168.0.218:33700,192.168.0.212:33700"); // 218 failover to 212
+ coinflip = true;
+ }
Context ic = new InitialContext(props);
Here is Ken's full blown specifications for the IIOP load balancing feature
being worked on (that doesn't exist yet...) As you can see, it's quite
involved.
> I have incorporated comments from Ken Saks on EJB and Vivek on SQE
> testing.
> This is version 0.5.
>
> Please review this, especially the last section and scheduling, and
> send me comments.
> When we start on this is currently unknown, due to a lot of other
> issues (especially the
> unrelated ORB escalation from Argela).
>
> Ken.
>
> Argela RFE
>
> version 0.5
>
> Argela
> (a telecom company in Turkey) has requested that Sun support
> per-request load balancing (PRLB) for stateless session beans. The
> current IIOP load balancing mechanism only works on an InitialContext
> basis: when a new InitialContext() is created, a load balancing
> decision is made, binding that InitialContext to a replica, until the
> replica fails, at which point all requests using that InitialContext
> fail over to a new replica.
>
> Argela's
> initial attempt to work with GFv2 was to create a new InitialContext
> for each request. That works, but has an unfortunate side effect. The
> InitialContext is created from a URL that contain multiple addresses in
> a fixed manner. These addresses (of the replicas) are always tried in
> the same order, so a failure has the unpleasant result of always
> waiting for the failed node to time out, which is unacceptable. This in
> fact bypasses the mechanism in place using GMS to prevent this sort of
> behavior.
>
> We
> have previously implemented a somewhat similar feature in a different
> context: support for hardware load balancing. However, that assumes the
> use of some sort of load balancing engine, and we are not assuming this
> in the Argela deployment.
>
> Design
>
> The basic outline is pretty simple:
>
> - Add a new CORBA policy and tagged component representing the
> policy of using per-request load balancing.
>
> - Modify
> the CORBA client code to look for the new policy. If the policy is
> present, modify the retry logic to go to the next endpoint in the IOR.
>
> - Modify the EJB deployment descriptor to add a proprietary
> per-request loadbalancing attribute to the XML descriptor
>
> - Modify EJB to pass the new CORBA policy to the ORB on deployment.
>
>
> The Policy and TaggedComponent
>
> -
>
>
> Create a new class PerRequestLoadBalancingPolicy
> in spi/extension. Model this on RequestPartitioningPolicy,
> with a boolean value to indicate whether or not to use PRLB.
>
>
> - Add a new constant REQUEST_PARTITIONING_POLICY = SUNVMCID.value +
> 5 to spi.orbutil.ORBConstants.
>
> -
>
>
> Add a new interface PerRequestLoadBalancingComponent
> to spi.ior.iiop. It has one method: boolean
> usePerRequestLoadBalancing().
>
>
> -
>
>
> Add a new class implementing the new component
> to impl.io.iiop. Again, model this on the RequestPartitioningComponentImpl
> in impl.ior.iiop.
>
>
> - Add a new constant TAG_REQUEST_PARTITIONING_ID =
> SUN_TAGGED_COMPONENT_ID_BASE + 5.
>
> - Add
> new methods makePerRequestLoadBalancingComponentFactory and
> makePerRequestLoadBalancingComponent to spi.ior.iiop.IIOPFactories.
>
> -
>
>
> Add code to ORBConfiguratorImpl.initIORFinders
> to add the tagged component factory to the tagged component
> IdentifiableFactoryFinder.
>
>
> -
>
>
> Add code in SocketOrChannelAcceptorImpl.makeIIOPProfileTemplate
> to check for a PerRequestLoadBalancing
> instance in the policies. If the policy is present, create and add the
> PerRequestLoadBalancing
> Tagged Component to the iiop profile template.
>
>
> This is very simple, and follows the existing work
> on RequestPartitioning,
> which is almost identical in impact on the code. This will take about 1
> day to implement and test
>
> The Client Side
>
> An EJB
> instance is represented (in GlassFish) by a dynamic stub, which
> contains a reference to a CorbaClientDelegateImpl. The
> CorbaClientDelegateImpl instance contains a CorbaContactInfoList, which
> represents the available endpoints (represented by CorbaContactInfo)
> which may be used to invoke a remote method on the object. The
> CorbaClientDelegateImpl.request method starts the invocation, and this
> is where the CorbaContactInfoList is obtained, through a call to
> CorbaContactInfoList.iterator(). Note that the CorbaContactInfoList
> contains the target IOR and so it has access to the PRLB tagged
> component, if it is present.
>
> Internally, the CorbaContactInfoList.iterator()
> method calls createContactInfoList(), which in turn calls
> addRemoteContactInfos( IOR effectiveTargetIOR, List<CorbaContactInfo>
> list ). This method makes use of an ORB SPI, the IORToSocketInfo
> interface, which defines a single method:
>
> public List<? extends SocketInfo> getSocketInfo(
> IOR ior, List<? extends SockectInfo> previous )
>
> The
> ORB has two versions of this interface: one in
> impl.transport.DefaultIORToSocketInfoImpl, and another implemented by
> impl.folb.ClientGroupManager.
> The details are not important for PRLB, but this does suggest that we
> should not implement PRLB in the IORToSocketInfo class (which is
> possible); instead, we can add handling for this directly in the
> CorbaContactInfoListImpl.iterator()
> method. There is also an implementation of IORToSocketInfo in GlassFish in
> com.sun.enterprise.iiop.IORToSocketInfoImpl, but that seems not to be
> used anywhere. The ClientGroupManager
> does use the csiv2SSLTaggedComponentHandler to get the CSIv2
> information, so this is probably just the old pre-IIOP FOLB version,
> and most likely the GlassFish
> code should be removed. In any case, we'll need to clean this up
> further in V3.
>
> Unfortunately
> the semantics of getSocketInfo is confusing. An implementation MAY
> chose to create a new list on every call, or re-use the previous list
> if it is non-null, so PRLB needs to accommodate either case. Generally
> the existing implementation re-use the existing list if it is not null,
> except in the case of CSIv2, which always constructs a new list (!).
> For now, PRLB will not work with CSIv2, just as IIOP FOLB does not at
> present work with CSIv2.
>
> Another detail that is important here is the
> effective target IOR in the CorbaContactInfoList.
> The target IOR is the IOR which was originally unmarshaled when the
> stub was constructed; the effective IOR starts out the same as the
> target IOR, but is updated by a location forward from IIOP, which calls
> back into the CorbaContactInfoList.setEffectiveTargetIOR
> to update the effective target IOR. The targetIOR is never changed, and
> plays a role in how retries are handled. The important point here is
> that setting the effective target IOR also clears
> effectiveTargetIORContactInfoList.
>
> Probably
> the easiest modification to make is to rotate the head to the tail of
> the effectiveTargetIORContactInfoList at the end of each call to the
> iterator method. The new method needs to look something like:
>
>
> function isnumbered(obj) {
> return obj.childNodes.length && obj.firstChild.childNodes.length &&
> obj.firstChild.firstChild.className == 'LineNumber';
> }
> function nformat(num,chrs,add) {
> var nlen = Math.max(0,chrs-(''+num).length), res = '';
> while (nlen>0) { res += ' '; nlen-- }
> return res+num+add;
> }
> function addnumber(did, nstart, nstep) {
> var c = document.getElementById(did), l = c.firstChild, n = 1;
> if (!isnumbered(c))
> if (typeof nstart == 'undefined') nstart = 1;
> if (typeof nstep == 'undefined') nstep = 1;
> n = nstart;
> while (l != null) {
> if (l.tagName == 'SPAN') {
> var s = document.createElement('SPAN');
> s.className = 'LineNumber'
> s.appendChild(document.createTextNode(nformat(n,4,' ')));
> n += nstep;
> if (l.childNodes.length)
> l.insertBefore(s, l.firstChild)
> else
> l.appendChild(s)
> }
> l = l.nextSibling;
> }
> return false;
> }
> function remnumber(did) {
> var c = document.getElementById(did), l = c.firstChild;
> if (isnumbered(c))
> while (l != null) {
> if (l.tagName == 'SPAN' && l.firstChild.className == 'LineNumber')
> l.removeChild(l.firstChild);
> l = l.nextSibling;
> }
> return false;
> }
> function togglenumber(did, nstart, nstep) {
> var c = document.getElementById(did);
> if (isnumbered(c)) {
> remnumber(did);
> } else {
> addnumber(did,nstart,nstep);
> }
> return false;
> }
> document.write('Toggle line numbers');
> Toggle line numbers
> 1 public synchronized Iterator<CorbaContactInfo> iterator() {
> 2 createContactInfoList() ;
> 3 Iterator<CorbaContactInfo> result = new CorbaContactInfoListIteratorImpl(
> orb, this,
> 4 primaryContactInfo, effectiveTargetIORContactInfoList ) ;
> 5 if (usePerRequestLoadBalancing) {
> 6 CorbaContactInfo head = effectiveTargetIORContactInfoList.remove(0) ;
> 7 effectiveTargetIORContactInfoList.add( head ) ;
> 8 }
> 9 }
>
>
> This has no effect on the rather complex retry logic
> that appears elsewhere in the request method, and especially in the
> CorbaContactRequestInfoIteratorImpl
> code. The iterator() method is only called on the FIRST attempt to
> invoke an endpoint; subsequent attempts use the same iterator (until
> success, unretryable error, or the iterator runs out of alternates).
> Also note that anything that clears or resets the
> effectiveTargetIORContactInfoList will cause the PRLB process to start
> over again with a new iterator, which is fine, as normally the list is
> created on demand and then lives as long as the CorbaContactInfoListImpl,
> which has the same lifetime as the the stub.
>
> One last point remains: we need to set the
> usePerRequestLoadBalancing flag. This is easy to do in the
> CorbaContactInfoListImpl.setEffectiveTargetIOR
> method, which is called whenever the effective IOR is changed, either
> when the list is created, or when a location forward is received.
> Basically all that is needed is to extract the PerRequestLoadBalancingComponent
> (if any) from the IOR's IIOPProfileTemplate, which indicates whether or
> not PRLB should be used. This will take around 1 day to implement.
>
> The EJB work
>
> A new minor sun-ejb-jar dtd revision, sun-ejb-jar_3_0-1.dtd,
> will be created. It will containa new optional boolean child element of
> the ejb element called per-request-load-balancing.
> If per-request-load-balancing is set to true
> for a stateless session bean, per request load balancingwill be enabled
> for invocations made through Remote EJB 2.x and 3.x client invocations
> on thatbean. If set to false or not set, per request load
> balancing will not be enabled for the bean. The
> per-request-load-balancing element only applies to stateless session
> beans. Use of the element on other bean types will result in a
> deployment error. This is also estimated at 1 day.
>
> Testing
>
> There are
> two kinds of testing needed here: unit testing in the ORB, and system
> testing. The ORB testing is fairly easy, given that we already have a
> framework for testing with multiple processes. System testing will be
> more difficult.
>
> Unit Testing
>
> Setup:
>
> - A client controller that simply invokes a method several thousand
> times.
>
> - An ORBD which provides the name service
>
> - Several server controllers which implement identical objects
>
> - A master program to control all of this
>
> Test cases:
>
> - Client
> makes N invocations to a PRLB object implemented on 4 servers; each
> server should be invoked N/4 times (or very close to that distribution)
>
>
> - Client makes N/4
> invocations per server to 4 servers; then 1 server is shut down.
> Thereafter Client makes M/3 invocations per server.
>
> - Client
> makes N/3 invocation per server to 3 servers; then 1 server is started
> up. Thereafter Client make M/4 invocations per server.
>
> These
> test cases can be developed starting from the existing CORBA lb test,
> which will need to be cloned to a new test and extensively modified.
> Only part of the logic of the existing test can be used, and I expect
> that this will take around 4 days.
>
> System Testing
>
> Basic outline:
>
> - Attempt
> to configure PRLB for a bean that is NOT a stateless session bean.
> Check that a reasonable error message is given on attempt to deploy the
> bean.
>
> - Configure a
> cluster with 4 instances. Deploy a PRLB stateless session bean to the
> cluster. Create 4 clients with 4 stateless session beans. Have each
> client run a large (at least 1000) number of times. Then test the
> following scenarios:
>
>
> - Run
> 4 client threads creating 4 stateless session beans. Check that all
> method calls are distributed evenly from each client to each instance.
>
> - Remove one instance from the cluster. Check that methd calls
> are distributed evenly across all 3 instances.
>
> - Add another instance to the cluster. Check that method calls
> are distributed evenly across all 5 instances.
>
>
> - Make sure that the existing IIOP FOLB tests continue to work when
> PRLB is NOT used.
>
> Basically
> we want to make sure that PRLB works, and that it does not cause any
> problems with existing IIOP FOLB tests, since much of the same code is
> shared in the ORB for these cases.
>
> The SQE estimate (from Vivekanandh Sedhumadhavan) is
> around 5 working days to implement and execute the above tests.
>
> Overall Schedule Estimates
>
>
> Item
>
>
> Time (working days)
>
>
> ORB policy support
>
>
> 1
>
>
> ORB client side support
>
>
> 1
>
>
> EJB support
>
>
> 1
>
>
> ORB unit testing
>
>
> 4
>
>
> SQE testing
>
>
> 5
>
>
> The total is 12 days. If we estimate the load factor
> to be 60%, that is 20 days, or 4 weeks. This works out to roughly:
>
>
> Phase
>
>
> Date
>
>
> Start
>
>
> TBD
>
>
> DEV and unit test done
>
>
> TBD+3 weeks
>
>
> Early Alpha release to Argela (if desired)
>
>
> TBD+3-4 weeks
>
>
> SQE testing for new feature ONLY done
>
>
> TBD+5 weeks
>
>
> Beta release to Argela
>
>
> TBD+5-6 weeks
>
>
> Complete full SQE test cycle; release official
> patch
>
>
> TBD+12 weeks