Memory Replication for Application Server 9.1 (Glassfish v.2)

Larry White

Mohamed Abdelaziz (JXTA)

Collaborative Effort

Larry White – HA architect/module owner (overall lead)

Mohamed Abdelaziz - JXTA tech lead

Jan Luehe – Web Container module owner

Mahesh Kannan – Ejb Container module owner

Shreedhar Ganapathy – GMS module owner

Special thanks to Jeet for his vision and steadfast support & to Bernard Traversat and Abhijit Kumar for insuring a productive collaboration

Agenda

Scope

Overview

Configuration

JXTA, Performance & Security

SPI

Q&A

Scope – problem

Up through appserver 8.2 (and beyond):

HADB provides high availability for

Http session state

Single Sign On state

Stateful (EJB) Session bean state

HADB not going open source into Glassfish

But still available via JES and can still be used in 9.1EE for “5-nines” availability

HADB weaknesses

heavy-weight, hard to administer, high TCO

Scope – problem

Need an open-source, lighter-weight replacement for HADB in Glassfish

Scope - solution

Memory replication

provide similar availability for

Http session state

Single Sign On state

Stateful (EJB) Session bean state

be fully feature compatible with current HADB availability story

Scope

Will address other requirements in course of remaining presentation

Agenda

Scope

Overview

Configuration

JXTA, Performance & Security

SPI

Q&A

Overview

Basic Terms

Instance – an app server instance (in this context a clustered instance)

Cluster shape – the list of instances plus the knowledge of where each instance is replicating

expected cluster shape

actual cluster shape

Cluster shape change – changing the shape of the cluster

Typically in response to various scenarios

Cluster Topology

Overview

Cluster Topology

Cluster shape

Ring – each instance replicates to one other instance

Pluggable Arranger – determines shape

This release: instances are arranged in aphabetical order based on instance name (“SimpleArranger”)

Fine for horizontal (one-instance per machine) clusters

For now: care must be taken by admin naming instances to maximize availability (see next slide)

Will be possible to use network metadata (from JXTA) to automatically arrange cluster shape to maximize HA (HAArranger is stretch goal)

Cluster Topology

Example: Maximize Availability on 4 node cluster on 2 machines

Overview

Typical Failover Scenario

Location transparency: fail over request can go to any instance in the cluster

2 Cases: Failover request lands on:

Case 1: instance with replica data: ownership taken, processing continues

Case 2: instance w/o replica data:

instance sends Self-Addressed-Stamped-Envelope (SASE) broadcast request

Instance with replica data (only) transfers data back to requestor and deletes its copy after an ack

JXTA makes this easy (Propagation communication channels are scoped within the “group” (i.e. the cluster members)

HTTP Session State Failover

HTTP Session State Failover

HTTP Session State Failover

HTTP Session State Failover

HTTP Session State Failover

Overview

Health Checking (passive & active)

Passive health checking

Errors caught while sending messages

Pipe closed events received when partner instance is stopped gracefully

GMS provides authoritative view of cluster membership useful in failure scenarios

Overview

Health Checking (passive & active)

Active health checking

to learn of an instance newly added to cluster and starting

to learn of instance failures; especially when instances not stopped gracefully

GMS will be responsible

Overview

Health Checking (instance failure/re-start)

Upon failure of replica instance

Instance that was replicating to the failed instance

continues normal request processing but suspends replication

readies & awaits re-start of failed instance

upon re-start, hand-shake

replication re-enabled

repair “under load” initiated in background

How long to wait for failed instance?

Default time – 30 minutes (might need this configurable) idea is enough time to re-start failed instance or know that you can't

Or sooner if asadmin remove-instance from the cluster

When you give up waiting, then cluster shape-change

Overview

Health Checking (instance failure/re-start)

When you give up waiting, then cluster shape-change

Overview

Cluster Shape Change

Similar mechanisms as for an instance stopping and re-starting

Reshaping for a removed instance

Following slides demonstrate this

Reshaping for an added instance

Similar procedure

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Cluster Shape Change Scenarios

Agenda

Scope

Overview

Configuration

JXTA, Performance & Security

SPI

Q&A

Configuration

Zero Configuration (default)

Port Range – if we can assume an 'owned' port-range fine otherwise, we will need to configure that in domain.xml

JXTA requires one port per instance

JXTA applying to IANA for 9701-9746 (available)

On multiple-instance machines, any port conflicts handled automatically (“virtual to physical address translation”)

<availability-service availability-enabled=”true” replicator-port-range=”9700-9999”>
.......
</availability-service>

Configuration

Zero Configuration (default)

Secure channels (false by default)

<availability-service availability-enabled=”true” secure-replication=”true”>
.......
</availability-service>

Configuration

Zero Configuration (continued)

We use only our familiar instance-names (e.g. i1, i2, etc) to call simple api's in JXTA to construct dynamically all needed artifacts

PeerID's

Pipe Advertisements

Pipes

GMS uses our cluster name in similar fashion to construct

Group ID

Configuration

Zero Configuration (continued)

Applications: wherever persistence-type=”HA”, can now use persistence-type=“replicated”

The configuration is the same

The semantics are the same

QA tests will require only this modification

Docs also require little change

HADB (when installed) still supported just as before

Agenda

Scope

Overview

Configuration

JXTA, Performance & Security

SPI

Q&A

Performance

Initial Trade 2 (HTTP) results:

Earlier results

No replication – 570 hits/sec

HADB – 330 hits/sec

Memory Replication – 410 hits/sec

28% degradation (exit criteria is 20%)

Still early days – more optimizations to come

Recent results:

No replication – 920 hits/sec

Memory replication – 716 hits/sec

22% degradation

JXTA Security

PKI-based

Pluggable KeyStore

“End-to-end” TLS secure channels

Each cluster can define its own membership and authentication policy

Scoped cluster traffic

Simple interface

No additional port needed

Single property to toggle secure channels

JXTA Security example

Use AppServer KeyStore

//during platform initialization

KeyStoreManager.setKeyStore(appKeyStore);

Toggle Security

pipeAdv.setType(PipeService.UnicastSecure);

socket = new JxtaSocket(cluster,

instance1ID,

pipeAdv,

30000,

// reliable connection

true);

JXTA Performance improvements

Current performance improvements :

Utilize direct exclusive channels whenever possible

Optimized message overhead

Generic improvement for general consumption

Improvements in progress

Eliminate message envelop overhead in direct mode

Utilize direct channels bidirectionally

Future performance improvements :

Mohamed and Larry joining Peregrine/Grizzly v2 effort

Utilize Grizzly V2 NIO libraries

Reduces threads-to-connection ratio

Remove one data copy operation

Utilize thread and direct buffer pools

JXTA Virtual Network

Simple addressing

Instance name encoded unique identifiers

Node

Cluster

Communication channels

Simplifies configuration

Traffic scoping

Dynamic route mapping

Mobility

Fault tolerance (dynamic route repair)

WAN support

JXTA Virtual Network

JXTA Virtual Network

Agenda

Scope

Overview

Configuration

JXTA, Performance & Security

SPI

Q&A

SPI for BackingStore

some basics here

see interface definitions in Appendix A for more details

SPI for BackingStore

Separate BackingStore Interface for:

Http session replication

Sso replication

Ejb SFSB replication

Default implementation is JXTA-based

Any configuration problems will result in default being used

One implementation for entire cluster

SPI for BackingStore

Basic points

Each implementation instance is servicing a separate component – i.e. web app context, SFSB container or virtual server

How this is done is an implementation detail

This information is injected into each implementation instance during the init method

It is not correct to assume that id's are guaranteed to be unique

In web container this is the case by spec

Sufficient 'metadata' is incorporated into the api's to ensure correct semantics for all containers

Not in Scope (this release)

Not a generic memory replication framework

This release tailored to the specific needs of web and ejb containers

Can plan for the future but will need specific requirements backed by clear use cases

Not transactional, not XA-compliant

Not on a WAN

Can be in future release

Possible Futures

Add functionality based on requirements

WAN based replication (doable with JXTA)

For our same use case (HTTP/SFSB)

Setting up tests would be interesting ;-)

A simple distributed key/value cache

Is this a solution looking for a problem or are there real potential users? If so, this could be implemented.

Binod has mentioned possible use by JBI but they may need ACID properties – ditto for the MQ team who need XA

Possible Futures

Add functionality based on requirements

Possible co-work with Database Technology Group

Transactional Semantics (with & w/o XA)

Database (record & typed field) semantics

Summary

Features

we have what we need

performance/scalability

good now

even better in future (incorporating Peregrine/Grizzly v2 enhancements)

Platform for future roadmap

Q & A

larry.white@sun.com

mohamed.abdelaziz@sun.com

Appendix A SPI

Configuration

Interface Definitions

BackingStore Configuration

<availability-service availability-enabled=”true” backing-store-bootstrap=”com.foo.MyBackingStoreBootstrapImpl”>
.......
</availability-service>

BackingStore Bootstrap & Factory

public interface BackingStoreBootstrap {

/* called during startup – do bootstrap work*/
public void start();
/* called during shutdown – do cleanup work*/
public void stop();
/*return factory used to get BackingStore implementations
public BackingStoreFactory getFactory();

} //BackingStoreBootstrap implementation needs a no-arg public constructor

public interface BackingStoreFactory {

/*return backing store impl for http session storage*/
public HttpBackingStore createHttpBackingStore();
/*return backing store impl for sso element storage*/
public SsoBackingStore createSsoBackingStore();
/*return backing store impl for ejb sfsb storage*/
public EjbBackingStore createEjbBackingStore();

}

BackingStore

public interface BackingStore {

public void initialize(Map replicationEnv);
public MetaData load(String id);
public void save(String id, MetaData value);
public void updateLastAccessTime(String id, long time);
public void remove(String id);
public void removeAll();
public void removeExpired();

}

public interface HttpBackingStore extends BackingStore {

public void saveAllAttributes(String id, HttpMetaData value);
public HttpMetaData loadAllAttributes(String id);

}

public interface EjbBackingStore extends BackingStore {

public void saveAll(EjbMetaData[] beanStates);

}

BackingStore

public interface SsoBackingStore extends BackingStore {

public void associate(String sessionId, SsoMetaData value);
public void removeInactiveSessions(String ssoId);

}

MetaData

public interface MetaData {

public long getVersion();
public long getLastAccessTime();
public long getMaxInactiveInterval();
public byte[] getState();

}

public interface HttpMetaData extends MetaData {

public ModifiedAttributeMetaData[] getAttributes();

}

public interface EjbMetaData extends MetaData {

public String getId();
public String getAppId();

}

MetaData

public interface ModifiedAttributeMetaData extends MetaData {

public String getAttrCommand; //”insert”, “update”, “delete”
public String getAttributeName();

}

public interface SsoMetaData extends MetaData {

public String getSsoId();
public String getAppId();
public String getUserName();
public String getSessionId();

}

Larry White

2006-08-23T11:48:10