The Customer Data Infrastructure (CDI) is a prime example of a Data-Intensive Application. Martin Kleppmann’s book Design Data-Intensive Applications is an excellent resource for understanding what a data-intensive application is. Essentially, CDI serves as an infrastructure for capturing, processing, and routing streams of events from applications. It should be noted that CDI differs from a customer data platform (CDP) within the customer data ecosystem. While both handle first-party data sets and data management, as well as data integration and pipelines, the focus of SaaS CDPs is on user interfaces for data-driven marketing teams. CDI, however, focuses on the componentry of customer data collection, ingestion, storage, and routing/integration.
Routing is an essential function of customer data infrastructure, sending real-time, event stream data from CDI data sources to various systems and applications on the cloud. Deduping is another challenge, ensuring that duplicate events are not sent to their destinations. This is a part of the series of engineering challenges faced at RudderStack.
The CDI consists of three main components: clients for data collection, the data plane for handling the data, and destinations for delivering it. The data plane is a distributed system, running on different servers that must provide fault tolerance and consistency guarantees even under failure conditions.
Handling failures in delivering data to destinations is crucial, as downstream teams use this data for critical functions such as creating actionable insights and metrics, driving customer engagement, executing campaign optimization, increasing customer retention, and customizing offers for individual customers. RudderStack aims to design a system that can gracefully handle these scenarios.
When handling events for a destination, a retrying mechanism can be implemented, with exponential back-off, to ensure that message failures do not cause loss of data or create unnecessary delay. The goal is to maintain order of events while gracefully handling failures.