Client Reroute

When the database fails over from one server to another, it is important to reroute clients to the new server. DB2 provides multiple methods for automated reroute.

Table of Contents

Automatic Client Reroute (ACR)

The DB2 Automatic Client Reroute (ACR) feature automatically reroutes clients to an alternate server when the primary server fails. It is the preferred reroute method.

ACR can be used between any two servers, not just HADR primary and standby. It is up to the administrator to set up replication to ensure that the two servers have the same data content. Other replication methods, such as Q-rep can be used to sync up the two servers.

How does it work?

Overview

  • User issues the " update alternate server " command on the primary database, specifying the standby as the alternate server.
    • It is recommended that you also issue the "update alternate server" command on the standby database, specifying the primary as the alternate server. This way, ACR will function smoothly after role switch or failover.
    • Only TCP client-server protocol is supported.
      • The "port" option only accepts port number. It does not accept service name.
    • The two servers must use the same client authentication method.
    • Use the server address for client-server communication (server hostname, dbm config parameter svcename) in "update alternate server" command. Do not use the HADR inter-server communication address (hadr_local_host, hadr_local_svc).
    • The "update alternate server" command is only supported on servers. If issued on a client installation, SQL1889W "The update alternate server request was ignored" message will be returned.
    • To clear alternate server , run the "update alternate server" command with "using hostname NULL port NULL" ("NULL" must be in uppercase).
    • To view alternate server , run the "db2 list database directory" command, you will see lines like (hostname and port will be shown as blank if alternate server is not defined):

Alternate server hostname            = gauss
Alternate server port number         = 1000

  • When a client connects to the primary databases, the alternate server information is propagated to the client.
    • Alternate server info is centrally controlled on the server side, making administration simple.
    • Client records alternate server info in memory for reconnection after abnormal connection termination.
    • Client records alternate server info on disk for starting future connections.
      • The recorded info is shared among all client processes on the same host.
      • When a client starts a connection to a server, it uses on-disk alternate server info to try multiple servers.
    • See " First connection Alternate Server " section below for bootstrapping the first connection on client side.
  • When an established connection ends abnormally, ACR will automatically try to reestablish the connection (no coding in application layer is needed).
    • ACR will alternately attempt to connect to the original server and the alternate server until a connection succeeds or a timeout is reached. Retry interval and number of retries can be configured on client side. See non Java ACR configuration and Java ACR configuration
    • If connection is reestablished (to original or alternate server), SQL30108N " A connection failed in an automatic client reroute environment. The transaction was rolled back. ... " is returned to application (for Java applications, the error can be exception -4498)
    • Seamless reroute (no error is returned at all) is possible in certain scenarios. See " Info Center Topics for ACR " below for more info. The main conditions are listed in non Java application programming for ACR and Java application programming for ACR . Additional info is spread in various topic.
    • Upon reconnection, the clients updates its alternate server list using a new list from the new server, if the new server also has server side alternate server info.
    • Transaction context is not preserved upon reconnection. Current transaction is rolled back.
      • Typically, applications resubmit the transaction.
      • An application may need to query data on the alternate server first before deciding to resubmit the transaction, because there is no guarantee that all transactions committed on the old server have been replicated to the new server (there can be data loss during the failover).
      • It is also possible that the last transaction before reroute is committed on the new server even if it is never reported as committed on the old server, because the old server or connection to old server can fail right after the transaction has been committed on the old server and replicated to the new server, but before the "commit success" message reaches the client.
    • ACR treats the two servers as alternate of each other. When failure to one occurs, it tries the other. So if alternate server fails, connections can be redirected back to the original server.
      • There is no automatic fail back (except with client affinity). Client stays with a server as long as connection to the server is good.
    • If connection cannot be reestablished within timeout, usual connection error SQL30081N " A communication error has been detected " is returned (for Java applications, the error can be exception -4499)
  • Both non Java and Java clients are supported.
  • ACR is not supported on Java DriverManager default connection (jdbc:default:connection). With default connection, the client actually runs in server memory space. No server network address (such as host name and port) is needed. Java default connection is not a TCP connection, therefore not supported by ACR.

Alternate Server Same as Primary Server

You can define alternate server to have the same address as the primary server. For example, primary server is on myserver.ibm.com, port 4000. You can run "db2 update alternate server for db myDB using hostname myserver.ibm.com port 4000" on the server. Such setup is used in the following scenarios:

  • Enable auto reconnect for transient network or server failure, even if no failover is involved.
  • Used in conjunction with virtual IP address (VIP) for enhanced rerouting. For example, myserver.ibm.com is already a VIP, you then define alternate server also on myserver.ibm.com.
    • This enables auto reroute of existing connections at failover time, because ACR will retry the VIP.
    • This reduces reroute time of new and existing connections, because ACR will not waste time trying the failed server (assuming VIP is remapped to the new server promptly during failover)
    • VIP with ACR is a recommended setup.
  • Used in conjunction with multi-home DNS to enable auto reroute of existing connections at failover time.

First Connection Alternate Server

A client does need to connect to the server once to get the alternate server info. If the client couldn't even connect to the server once, and the server goes down, the client cannot be rerouted to the alternate server. To deal with such scenario, alternate servers can be specified on the client side. Notes:

  • Client side alternate server is only used for first connection . First connection is defined as "client explicitly requests connecting to a server", to differentiate from ACR initiated auto reconnection after abnormal connection termination.
  • Client side alternate server overrides server side alternate server for first connection. During first connection, server side alternate server is ignored if client side alternate server is specified.
  • Server side alternate server is preferred due to its simplicity.
  • For reconnection after abnormal connection termination, server side alternate server is always used.

To configure client side first connect alternate server:

  • For non Java clients, use enableAlternateServerListFirstConnect and alternateserverlist fields in the db2dsdriver.cfg file. See also non Java ACR configuration
  • For Java clients, use clientRerouteAlternateServerName and clientRerouteAlternatePortNumber properties for the connection. See also Java ACR configuration

Alternate Group

Alternate group is extended ACR. With alternate groups, servers are managed in groups. Each group can include multiple servers (multiple host:port addresses). Alternate group is typically used in pureScale and DPF environment, where a group maps to a cluster.

  • More than one alternate groups can be defined, unless the server is DB2 for z/OS, which supports only one alternate group.
  • Reconnection favors servers in the same group.
  • Alternate group is configured on client side. It is harder to manage than standard ACR.

See Info Center for

Client Affinity

Client affinity is extended ACR. With client affinity:

  • Multiple alternate servers can be defined.
  • Preference among the servers can be defined.
  • Failback to primary server can be optionally enabled.
  • Client affinity overrides alternate group and standard ACR. When Client Affinity is enabled, alternate groups and server side alternate servers are ignored.
  • Client affinity is configured on client side. It is harder to manage than standard ACR.

See Info Center for

  • Non Java client (expand in navigation pane to see "Configuration" and "Example" sub-topics)
  • Java client (expand in navigation pane to see "Configuration" and "Example" sub-topics).

Info Center Topics for ACR

Server Side Topics

On server side, you just need to issue the " update alternate server " command. It is recommended to issue the command on both servers to make them alternate of each other.

Client Side Topics

ACR is enabled by default on client side. Usually, client side configuration is not needed to use ACR. Client side configuration is needed only for:

  • Disabling ACR for client (do not use ACR even if alternate server is defined on server side)
  • Fine tuning ACR client behavior (such as connection retry interval and number of retries)
  • Configuring first connect alternate server.
  • Configuring alternate group.
  • Configuring client affinity.

Non Java Client

Java Client

Usage Notes

HADR Admin Commands

The HADR commands (start/stop/takeover HADR) connects to the database via the usual client-server connection. If the database alias used in an HADR command has ACR enabled, you don't have control on whether the command is directed to the primary or the standby. Thus HADR commands should run on a database alias that has no ACR (or any other reroute method). For admin purpose, you may define an alias with ACR for application access, and an alias with no ACR for HADR commands.

Reads on Standby

ACR considers a reads-on-standby enabled standby database "connectable". If ACR is enabled between the primary database and a reads on standby enabled standby database, both databases will appear available and ACR may randomly direct a client to one of databases. Thus in reads on standby environment, do not enable ACR between primary and standby servers. Instead, use VIP (Virtual IP) for client reroute instead. See white paper Enabling continuous access to read on standby databases using Virtual IP addresses , which describes how to setup a VIP for read/write access (primary server) and another VIP for read-only access (standby server).

Multiple Standbys

With ACR, only one alternate server can be specified on server side. In the case of multiple standby HADR systems, the common practice is to configure the primary and the principal standby as alternate server of each other, because failover is more likely to happen between these two servers. Alternatively you may use other reroute methods, such as Virtual IP , multi-home DNS , or client side alternate server configuration for ACR ( client affinity , alternate group ) to support multiple alternate servers.

PureScale Systems

The pureScale white papers from More Readings are excellent resources for ACR and WLB (workload balancing) on pureScale.

Automatic Client Reroute in pureScale

Reroute within a cluster

In a pureScale system, you only need to configure a client to connect to one member of the server cluster, using that member's host and port address. When the client connects to the member, the member returns the cluster member list as alternate servers (in the context of ACR, each pureScale member is considered a "server" because it has its own host:port TCP address). If "update alternate server" command was run on the server, the server sends the user specified alternate server in addition to the member list. The member list and user specified alternate server are persisted to disk using the usual ACR alternate server persistence mechanism. Thus subsequent connections will try other members if the configured member is not available. Although first connection alternate servers can be configured on the client side to list multiple members, server side alternate server is preferred due to its simplicity.

Because ACR is enabled by default on client side, failover among members on the same cluster will just work without user configuration. ACR reconnection favors members in the same cluster. When connection to a member fails, ACR will first try reconnecting to members in the same cluster. If it cannot connect to any member in the cluster, it will try the user specified alternate server (if defined).

Reroute between clusters

Server side user specified alternate server can only include a single TCP address (host and port pair). If the alternate server is from a pureScale cluster, you can only specify one member's address from that cluster. If the specified member is offline, clients cannot be rerouted to the alternate cluster when the primary cluster fails. Other reroute methods, such as Virtual IP , multi-home DNS , or client side alternate server configuration for ACR ( client affinity , alternate group ) can be used to list multiple members from the alternate cluster. .

Workload Balancing in pureScale

While ACR can redirect clients from a failed member to another member, it does not balance loads on the members. For example, if clients are configured to connect to member 0 of a cluster, clients can be rerouted to other members if member 0 fails. But if member 0 is available, it would be overloaded without load balancing. A related DB2 feature, workload balancing (WLB) balances load across members. With WLB, server sends cluster member load information to the client periodically. The client uses this information to submit workload to less busy members to even the load. Load balancing can happen on connection or transaction level:

  • Non Java clients support both connection and transaction level balancing
    • Connection level balancing is enabled by default
    • Transaction level balancing is disabled by default
  • Java clients only support transaction level balancing
    • Transaction level balancing is disabled by default
    • Only type 4 connection is supported.

Info center topics for WLB:

Non Java Client

Java Client

Member Subsetting

Starting in DB2 V10.5, pureScale members can be divided into subsets for workload management with finer granularity.

Other Client Reroute Methods

VIP (Virtual IP)

In this method, the user defines a virtual IP that maps to the currently available database server. Clients simply connect to this VIP. Upon failover, network configuration is updated to maps the VIP to the new primary server. Clients can then transparently connects to the new primary server using the VIP. DB2 integrated cluster manager (TSA) can automatically update the VIP upon failover. See also white paper Automated cluster controlled HADR configuration setup using the IBM DB2 high availability instance configuration utility (db2haicu) , and cluster managers . Notes:

  • You must use the same port on all servers as VIP does nothing about port.
  • After failover, new connections will be rerouted to new primary database.
  • Unless ACR is also enabled, existing connections at failover time will still get connection failure error ( SQL30081N " A communication error has been detected ").
  • Using VIP with ACR together is recommended. This setup supports auto reroute of existing connections at failover time and reduces reroute time. See Alternate Server Same as Primary Server

Multi-home DNS

In this method, a DNS entry is created to list the IP address of a database server and its alternate servers (more than one alternate server can be listed). For example, the entry "myserver.ibm.com" maps to two IP addresses, one for HADR primary database host and one for the standby host. Clients are programmed to connect to "myserver.ibm.com". When a client makes the connection, it will retrieve the IP list from DNS and try the servers one by one until it establishes a TCP connection to one of the listed servers.

  • Both Java and non Java clients support multi-home DNS for alternate servers.
  • After failover, new connections will be rerouted to new primary database.
  • Unless ACR is also enabled, existing connections at failover time will still get connection failure error ( SQL30081N " A communication error has been detected ").
  • Using multi-home DNS with ACR together is recommended. This setup supports auto reroute of existing connections at failover time. See Alternate Server Same as Primary Server

Client Reroute Best Practices

  • Configure a virtual IP to point to the current primary server.
    • In pureScale systems, the VIP can point to an active member on the current primary cluster
    • VIP makes reroute fast because clients need not waste time trying failed servers
    • Use VIP together with ACR to support rerouting existing connections at failover time.
  • Run "update alternate server" commands on all servers for symmetric configuration.
    • With VIP, just run "update alternate server for db myDB using <VIP> port <serverPort>" on all servers. This setup supports more than one alternate server.
  • Use reroute enabled database aliases for application access. Use no reroute aliases for admin commands such as start/stop/takeover HADR.

More Readings