About MirrorMaker 2.0

MirrorMaker 2.0 is a capability within Apache Kafka that is used for copying (mirroring) topics from one Kafka cluster to another. MirrorMaker 2.0 can be used for a wide range of use cases such as data distribution, to disaster recovery, and infrastructure maintenance.

Since MirrorMaker 2.0 is a part of Apache Kafka, it is also part of Event Streams, and is fully supported.

Note: Also see geo-replication in Event Streams, which provides a simplified way of configuring MirrorMaker 2.0 for a subset of use cases.

MirrorMaker 2.0 makes use of the Kafka Connect framework, and offers dynamic configuration changes, offset synchronization, bidirectional mirroring, and improved performance.

Offset synchronization

An important feature of MirrorMaker 2.0 is the offset synchronization.

When topics are mirrored to another cluster, the event offset for a given event in the origin topic will be different to the event in the destination topic. This can occur due to records that are deleted from the origin topic due to retention policies, or because of the control records that are created when using transactions. The issue is that if a consumer moves from the origin to the destination cluster without considering these offset differences, it will duplicate, or worse, loose events.

MirrorMaker 2.0 handles these offset differences by storing a mapping between the offsets in the origin and destination clusters. It then uses this mapping to correctly populate the consumer offsets topic on the destination cluster. When consumers start up against the destination cluster, they will begin reading from the correct point in the topic’s event stream.

MirrorMaker 2.0 also synchronizes topic partitioning and access control lists, to ensure the destination cluster is as similar as possible to the origin cluster.

Reference

The following blog posts explain how to configure MirrorMaker 2.0 for its most common use cases.

Data distribution:

Disaster recovery:

Infrastructure maintenance: