# CMI (Clustered Method Invocation)

## Introduction

CMI is the protocol cluster for JOnAS ensuring:

- the jndi high availability through the registry replication and the multi-target lookup
- the load-balancing and fail-over at the EJB level (RMI flow) through the CMI cluster stub
  - for the Home interface of the EJB's SSB, SFSB, EB
  - for the Remote interface of the EJB's SSB
- the SFSB high availability with the JOnAS HA service

## Getting started with the CMI Protocol

CMI can be enabled in JOnAS by:

- setting the cmi protocol in the $JONAS_BASE/conf/carol.properties file
- compiling an application with the CMI protocol

## Clustered objects and CMI Registry

CMI brings its own registry for implementing the jndi replication. Each CMI registry instance over the cluster contains two areas:

- a local area for hosting the local objects that must not be replicated
- a distributed area hosting the global objects (cluster objects) that must be replicated

When an object is registered in the registry, the routing to the local or global area is done according to:

- the type of the object: only the remote objects are replicated
- the presence of the 'CMI class' having the same name as the object class with the suffix '_CMI' and inherited from the org.objectweb.carol.cmi.Distributor class
- one of the two following methods returning true:
  - equivAtBind() if the replication has to take place at the bind() time. For example, this could be the case for the EJB Remote Home objects.
  - equivAtExport() if the replication has to take place at the export() time. Typically, this could be the case for the SSB Remote objects.

The entries of the distributed area are lists providing the ability, for example, to gather several stubs for the the same jndi-name and thus to return a stubs list.

## JNDI HA

### Registry Replication

CMI relies on JGroups group-communication protocol for ensuring the global registry replication. The parameters are gathered in the:

- $JONAS_BASE/conf/carol.properties for specifying the JGroups configuration file name and the JGroups group name.
- $JONAS_BASE/conf/jgroups-cmi.xml file for the settings of the jgroups protocol stack. By default, the JGroups configuration uses the UDP protocol and the multicast IP for broadcasting the registry updates. A TCP-based stack can be used in a network environment that does not allow the use of multicast IP or when a cluster is distributed over a WAN.

All the members of a cluster share the same JGroups configuration.

If several cluster partitions are required over a single LAN, several JGroups configurations must be configured with different values for the following parameters:

- JGroups group name
- JGroups multicast address
- JGroups multicast port

When a new node appears in the cluster, its registry content is synchronized automatically.

When a node disappears, JGroups notifies the other's member of the node leaving and the registry entries related to this node are removed.

### Registry Fail-over

On the client side, the high availability of the registry is provided by the capability to set several JOnAS instances in the registry url. At the lookup time, the client chooses (round-robin algorithm) one of the available servers to get the home stub. If the server fails, the request is sent to another server. The CMI url registry is specified in the $JONAS_BASE/conf/carol.properties file using the following syntax:

    carol.cmi.url=cmi://server1:port1[,server2:port2...]

## CMI Cluster Stub or Cluster-aware Stub

Load-balancing and fail-over on the client side are provided through cluster-aware stubs. These stubs are generated on the fly through ASM and rely on:

- the RMI/JRMP protocol for communicating with the server side,
- the CMI class associated with the EJB for the load-balancing and fail-over logic.

### Cluster Map on the Client Side

The CMI cluster stub handles a cluster map on the client side. The CMI cluster stub is created:

- at lookup time for the SSB, SFSB, EB Home objects if the retrieved object is replicated and located in the global registry
- at create time for the SSB Remote objects if the retrieved object is replicated and located in the global registry

In these two cases, the call gets a stubs list from the global registry and the CMI cluster stub updates the local cluster map. Afterwards, the local cluster map can be updated dynamically during the invocation of the business methods calls (through the HA interceptors) when a new view is detected in the cluster.

If a communication error with a server occurs during a remote call invocation, the server is removed from the cluster map.

### CMI Class

CMI classes are generated by GenIC when compiling with the protocol CMI. They are built from the velocity templates located in $JONAS_ROOT/templates/genic directory. By default the templates used are:

- ClusterHomeDistributor.vm for the home interface of the SSB, SFSB, EB
- ClusterRemoteSLSBDistributor.vm for the remote interface of the SSB

The templates inherit the org.objectweb.carol.cmi.Distributor class and contain the following methods:

- choose(method, params): this method is called by the CMI cluster stub at each remote method call to choose the next stub to use for invoking the call. By default, this method implements the weighted round-robin algorithm with a local preference, i.e., if there is a collocated server, it will be selected in priority. The servers factors are set for each JOnAS instance in the $JONAS_BASE/conf/carol.properties file.Below is given the code of the choose() method implementing a simple RR algorithm :

          public StubData choose(Method method, Object[] parameters) throws NoServerException {

              Set stubs = getCurrentState();
              if (lastSet != stubs) {
                  lastSet = stubs;
                  rr.update(stubs);
              }

              return rr.get();
          }

- onException(method, params, stub, exception): this method is called by the CMI cluster stub when an exception occurs and returns a decision of RETRY, RETURN, or THROW for indicating the behavior to the CMI smart stub. By default, this method returns a RETRY decision if the exception is related to a network error, and otherwise it returns a THROW decision. Below is given the code of the mustFailover() method that is used by default for analysing the exception and deciding if the fail-over must occur :

           protected static boolean mustFailover(Exception ex) {
              if (ex instanceof UnmarshalException) {
                  Throwable cause = ((UnmarshalException) ex).getCause();
                  if (( cause instanceof EOFException) ||
                          (cause instanceof SocketException)) {
                      return true;
                  }
              }

              if ((ex instanceof ConnectException) ||
                      (ex instanceof ConnectIOException) ||
                      (ex instanceof NoSuchObjectException)) {
                    return true;
              }
             return false;
           }

- onReturn(method, params, stub, returnValue): this method is called by the CMI cluster stub after having invoked the call and before returning to the application. This call back enables implementation of a post-processing on the return value. By default, this method does nothing.

### Customizing the Load-balancing and Fail-over Logic

The user has the ability to customize the load-balancing and fail-over logic for each EJB by specifying the velocity template to use in the JOnAS- specific descriptor on deployment of the ejb-jar file. The XML elements are:

    <cluster-home-distributor>MyHomeDistributor.vm</cluster-home-distributor>

    <cluster-remote-distributor>MyRemoteDistributor.vm</cluster-remote-distributor>

If not set, the default velocity templates are used.

If set with the value 'disabled', the CMI classes are not generated and the EJB will not be distributed.

If set with a file name, this file must be located in the $JONAS_ROOT/templates/genic directory.

The 'cluster-home-distributor' element is valid for the SSB, SFSB and EB.

The 'cluster-remote-distributor' element is valid for the SSB.

## High Availability with Horizontal Replication

Stateful session beans (SFSBs) can be replicated since JOnAS 4.7 in order to provide high availability in the case of failures in clustered environments. A new service called High Availability (HA) has been included in JOnAS to provide replication mechanishms. JOnAS HA also requires the cluster method invocation (CMI) protocol.

Compared to JOnAS 4.7, JOnAS 4.8 implements a new replication algorithm based on a horizontal replication approach. The algorithm improves the algorithm implemented for JOnAS 4.7 with the following enhancements:

- Replication of SFSBs with references to EBs: The algorithm can replicate SFSBs that reference EB by means of both, local or remote interfaces.
- Transaction awareness: The algorithm is transaction aware, meaning that the state is not replicated if the transaction aborts.
- Exactly-once semantics: Each transaction is committed exactly once at the DB if the client does not fail. If the client fails, each transaction is committed at most once at the DB

### EJB replication Description

#### Update-everywhere mode

JOnAS implements an update-everywhere replication protocol according to the database replication terminology (See the J. Gray et al.'s paper ''The dangers of replication and a solution'' in proceedings of the ACM SIGMOD 96's conference, Canada). In this protocol, a client can connect to any server. When the client calls the create() method on the SFSB's Home interface, the server the client connects to is selected following a round-robin scheme. All the requests from the client to the SFSB will be processed by this server until the client calls the remove() method on the remote interface. The rest of the servers will act as backups for that client. Before sending the response to the client, the SFSB's state is sent to the backups.

If the server fails, another server among the backups will be selected to serve the client requests, first restoring the current state of the SFSBs from the state information stored in the HA local service. From this point on, this server will receive the new client requests.

The supported replication scenarios are shown in the following figure:

#### Transaction aware fail-over

The horizontal approach aims to guarantee that the transactions are kept consistent when a fail-over occurs. They are either aborted or restored for ensuring the exactly-once semantics. During a fail-over, the new primary uses a special table in the database for storing the transaction identifier and enabling to find out if the transaction was committed or not.

- If the transaction is aborted due to the primary failure, then the new primary will not find the transaction identifier in the special table. The request will be replayed.
- If the transaction is committed, then the new primary will find the transaction identifier, which means that the transaction was committed. The request won't be replayed; the replicated result is returned.

Beyond the SFSB replication, the algorithm enables the building of applications (stateful or stateless) with a high level of reliability and integrity.

### Configuring JOnAS for EJB Replication

The High Availability (HA) service is required in JOnAS in order to replicate SFSBs. The HA service must be included in the list of available services in JOnAS. This is done in the jonas.properties file placed in $JONAS_BASE/conf.

    ...
    jonas.services registry,jmx,jtm,db,dbm,security,resource,ejb,ws,web,ear,ha
    ...

The HA service must also be configured in the jonas.properties file:

    ...
    jonas.service.ha.class org.objectweb.jonas.ha.HaServiceImpl
    jonas.service.ha.gcl jgroups
    ...

The HA service uses JGroups as a group communication layer (GCL). JGroups behavior is specified by means of a stack of properties configured through an XML file (See JGroups documentation for more information: http://www.jgroups.org). The default configuration of the HA service uses the $JONAS_BASE/conf/jgroups-ha.xml file and the sfsb-rep group name. The HA service can be told to use a particular stack configuration or a particular group name by modifying the following lines in jonas.properties:

    ...
    jonas.service.ha.jgroups.conf jgroups-ha.xml
    jonas.service.ha.jgroups.groupname jonas-rep
    ...

Finally, the CMI protocol must be specified in the carol.properties file in $JONAS_BASE/conf:

    ...
    carol.protocols=cmi...
    ...

### Transaction Table Configuration

The new horizontal replication algorithm uses a database table to keep track of current running transactions. This table is accessed from the new elected node during fail-over to detect whether or not the current transaction committed at the former local node, ensuring exactly-once semantics. The table contains only one column: the transaction identifier (txid).

In JOnAS 4.8 this table must be created manually with the following SQL command:

    create TABLE ha_transactions (txid varchar(60));

This table should be located preferably in the database used by the replicated application, but it is not mandatory. If the table is not created in the database used by the replicated application, it is necessary to configure a new datasource for the database that contains this transaction table. This datasource must be configured to use the serializable transaction isolation level.

The database that holds the transaction table is accessed by the replication service with the JNDI name configured in jonas.properties.

    ...
    jonas.service.ha.datasource tx_table_ds
    ...

### Configuring Garbage Collection

Due to the fact that the replication algorithm stores information associated with clients' transactions and that the server is not notified when a client dies, the HA service might have been storing unnecessary replication information over time. In order to automatically clean this unnecessary replication information, the HA service includes a garbage collection mechanism. It is possible to configure the number of seconds the system waits to execute this mechanism by changing the following property in the jonas.properties file:

    ...
    jonas.service.ha.timeout 600
    ...

### Configuring an Application for Replication

#### jonas-ejb-jar.xml

In order to configure an application for replication, the <cluster-replicated/> element must be added to the bean definition of every bean requiring high availability in the jonas-ejb-jar.xml deployment descriptor file. This element can have two possible values: true or false (default value). In addition, if the programmer wants to change the behavior of the CMI stubs (e.g., the server selection policy), it is possible to specify different distributor implementations by means of <cluster-home-distributor/> and <cluster-remote-distributor/> elements. In this case, the value corresponds to the .vm file that implements the distributor in its home and remote parts respectively. If the <cluster-replicated/> element is present without the <cluster-*-distributor/> elements, the default values are used (ClusterHomeSFSBRepDistributor.vm and ClusterRemoteSFSBRepDistributor.vm).

The following is an example description for a replicated SFSB in jonas-ejb-jar.xml file:

    ...
    <jonas-session>
       <ejb-name>DummySFSB</ejb-name>
       <jndi-name>DummySFSB</jndi-name>
       ...
       <cluster-replicated>true</cluster-replicated>
       <cluster-home-distributor>Dummy_HomeDistributor.vm</cluster-home-distributor>
       <cluster-remote-distributor>Dummy_RemoteDistributor.vm</cluster-remote-distributor>
    </jonas-session>
    ...

The <cluster-replicated/> element can also be set in the SSB or EB for

- enabling the transaction checking mechanism ensuring the exactly-once semantic at fail-over
- supporting the EB references replication

Note: When set in the SSB, the mechanism inhibits the load-balancing at the remote interface. After the home create() method call, all the requests are sent to the same instance.

#### Entity Beans lock policy

The lock policy for the Entity Beans in a replicated application must be configured as database in the jonas-ejb-jar.xml deployment descriptor file.

The following is an example description for a replicated EB in the jonas-ejb-jar.xml:

    ...
    <jonas-entity>
        <ejb-name>MyEntitySLR</ejb-name>
        <jndi-name>MyEntityHome</jndi-name>
        <cluster-replicated>true</cluster-replicated>
        <shared>true</shared>
        <jdbc-mapping>
            <jndi-name>example_ds</jndi-name>
        </jdbc-mapping>
        <lock-policy>database</lock-policy>
    </jonas-entity>
    ...

#### Datasource used by the application

The datasources used by replicated applications must be configured to use the serializable transaction isolation level.

The following is an example for a datasource configuration file for the Postgres DBMS:

    ...
    datasource.name         example_ds
    datasource.url          jdbc:postgresql://xxx.xxx.xxx.xxx:xxxx/database
    datasource.classname    org.postgresql.Driver
    datasource.username     jonas
    datasource.password
    datasource.mapper       rdb.postgres
    datasource.isolationlevel       serializable
    ...

Finally, when compiling the application that includes the replicated beans, the CMI protocol must be specified in order to generate the classes that include the replication logic.

### Status and Management Information in Admin. Console for the HA Service's Replication Algorithm

The JOnAS administration console offers several items of information about the HA service's replication algorithm and allows the configuring of several parameters related to its behaviour. The related information and parameters include:

- The name of the service.
- The binded name for the MBean. The name can be changed.
- The number of replicated messages sent by the algorithm to the cluster's replicas.
- The average size of the replicated messages sent.
- The total size of the replicated messages sent.
- The current JGroups configuration file name used.
- The current timeout established to clean in memory information related to SFSBs required by the algorithm. When this timeout expires, the information is garbage-collected. This avoids increasing the memory used by the algorithm. The administrator can set a different timeout if required.
- The datasource name required by the algorithm to keep track of current running transactions (See Transaction Table Configuration section above). The default datasource is setted through the "jonas.service.ha.datasource" parameter in the "jonas.properties" configuration file, but the administrator can configure different datasources and can set here, the name of the one that will be used by the algorithm, once JOnAS has started.

  NOTE: It is recomended to not change the Datasource once the HA service is running.