Monitoring Cloudera DataFlow Deployments With Prometheus and Grafana

By neub9
2 Min Read

HTML Content:

Monitoring NiFi Metrics in Grafana with CDF-PC

Monitoring NiFi Metrics in Grafana with CDF-PC

Posted in Technical | January 17, 2024

Cloudera DataFlow for the Public Cloud (CDF-PC) is a complete self-service streaming data capture and movement platform based on Apache NiFi. It allows developers to interactively design data flows in a drag and drop designer, which can be deployed as continuously running, auto-scaling flow deployments or event-driven serverless functions. CDF-PC comes with a monitoring dashboard out of the box for data flow health and performance monitoring. Key performance indicators (KPIs) and associated alerts help customers monitor what matters for their use cases.

Many organizations have invested in central monitoring and observability tools such as Prometheus and Grafana and are looking for ways to integrate key data flow metrics into their existing architecture.

In this blog we will dive into how CDF-PC’s support for NiFi reporting tasks can be used to monitor key metrics in Prometheus and Grafana.

Target architecture: connecting the pieces

The key to insightful Grafana dashboards is having access to relevant application metrics. In our case, these are NiFi metrics of our flow deployment. We therefore need to be able to expose NiFi metrics for Prometheus so it can scrape them before we can build dashboards in Grafana. CDF-PC’s support for Prometheus reporting tasks and inbound connections allows Prometheus to scrape metrics in real time. Once the metrics are in Prometheus, querying it and building dashboards on top of it in Grafana is straightforward. So let’s take a closer look at how we get from having metrics in our flow deployment to a fully featured Grafana dashboard by implementing the target architecture.

Target Architecture

Share This Article
Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *