Replication is a vital feature in distributed systems that addresses issues related to fault tolerance, high availability, load balancing, scalability, data locality, network efficiency, and data durability. It serves as the foundation for building robust and dependable distributed architectures. Having multiple options for replication, such as normal and prefixless replication, is important because each solution has its own advantages.
Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault-tolerant, scalable, and robust cross-cluster Kafka topic replication. SRM efficiently replicates data and ensures that topic properties are synchronized across clusters. It allows for dynamic enablement of replication for topics and consumer groups. Additionally, SRM includes custom extensions for installation, management, and monitoring, making it a complete replication solution for mission-critical workloads.
Kafka, as an event streaming component, can be applied to a wide variety of use cases, and SRM provides cross-cluster Kafka topic replication to enhance fault tolerance and robustness. SRM is based on the improved version of Mirror Maker (MM1), called Mirror Maker 2 (MM2), addressing several limitations of MM1.
To make replication more user-friendly, Cloudera introduced a new MM1-compatible mode in SRM, which supports prefixless replication starting with Cloudera Data Platform (CDP) Private Cloud Base 7.1.9. This allows for cluster migration workloads from deprecated MM1 to SRM without altering the replicated topic names.
The naming of the replicated topics is defined by the replication policy that SRM is configured to use. By default, SRM uses the DefaultReplicationPolicy, which prefixes the names of replicated topics with the source cluster alias. However, the IdentityReplicationPolicy does not change topic names during replication. Replication monitoring limitations associated with this policy were addressed in the CDP Private Cloud Base 7.1.9.
SRM relies on replication policy and topic naming conventions for remote topic discovery, which has its limitations. To improve this, SRM 7.1.9 introduced a new feature that writes updates to an internal, compacted topic, making it independent of the replication policy in use.
From CDP 7.1.9, SRM supports prefixless replication, replicating topics’ names exactly as on the source cluster. This is useful for topic aggregation use cases. To enable prefixless replication, the “Enable Prefixless Replication” property in the SRM service configuration must be selected.
While prefixless replication has its advantages, it does have limitations. For instance, it does not support replication loop detection, and all SRM services must use the same replication policy, whether it be prefixless or prefixed. Clusters connected by replication flows should only use one ReplicationPolicy, lest replications become mixed up.
Overall, SRM’s new features and improvements make it a robust solution for cross-cluster Kafka topic replication.