Monitoring Kafka cluster health

Monitoring the health of your Kafka cluster helps to verify that your operations are running smoothly. The Event Streams UI includes a preconfigured dashboard that monitors Kafka data.

Event Streams also provides a number of ways to export metrics from your Kafka brokers to external monitoring and logging applications. These metrics are useful indicators of the health of the cluster, and can provide warnings of potential problems. The following sections provide an overview of the available options.

For information about the health of your topics, check the producer activity dashboard.

JMX Exporter

You can use Event Streams to collect JMX metrics from Kafka brokers, ZooKeeper nodes, and Kafka Connect nodes, and export them to Prometheus.

For an example of how to configure the JMX exporter, see configuring the JMX Exporter

Kafka Exporter

You can use Event Streams to export metrics to Prometheus. These metrics are otherwise only accessible through the Kafka command line tools. This allows topic metrics such as consumer group lag to be collected.

For an example of how to configure a Kafka Exporter, see configuring the Kafka Exporter.

JmxTrans

JmxTrans can be used to push JMX metrics from Kafka brokers to external applications or databases. For more information, see configuring JmxTrans.

Grafana

You can use dashboards in the Grafana service to monitor your Event Streams instance for health and performance of your Kafka clusters.

Viewing installed Grafana dashboards

To view the Event Streams Grafana dashboards, follow these steps:

  1. Log in to your IBM Cloud Platform foundational services management console as an administrator. For more information, see the IBM Cloud Platform foundational services documentation.
  2. Navigate to the IBM Cloud Pak foundational services console homepage.
  3. Click the hamburger icon in the top left.
  4. Expand Monitor Health.
  5. Click the Monitoring in the expanded menu to open the Grafana homepage.
  6. Click the user icon in the bottom left corner to open the user profile page.
  7. In the Organizations table, find the namespace where you installed the Event Streams monitoringdashboard custom resource, and switch the user profile to that namespace.
  8. Hover over the Dashboards on the left and click Manage.
  9. Click on the dashboard you want to view in the Dashboard table.

Ensure you select your namespace, cluster name, and other filters at the top of the dashboard to view the required information.

Kibana

Create dashboards in the Kibana service that is provided by the OpenShift Container Platform cluster logging, and use the dashboards to monitor for specific errors in the logs and set up alerts for when a number of errors occur over a period of time in your Event Streams instance.

To install the Event Streams Kibana dashboards, follow these steps:

  1. Ensure you have cluster logging installed.

  2. Download the JSON file that includes the example Kibana dashboards for Event Streams from GitHub.

  3. Navigate to the Kibana homepage on your cluster.

    For IBM Cloud Pak foundational services: Click the hamburger icon in the top left and then expand Monitor Health. Then click Logging to open the Kibana homepage.

    For OpenShift Container Platform cluster logging stack: Log in to the OpenShift Container Platform web console using your login credentials. Then follow the instructions to navigate to cluster logging’s Kibana homepage.

  4. Click the Management tab on the left.
  5. Click the Saved Objects.
  6. Click the Import icon and navigate to the JSON file that includes the example Kibana dashboards for Event Streams you have downloaded.
  7. Click the Dashboard tab on the left to view the downloaded dashboards.

Other Monitoring Tools

You can also use external monitoring tools to monitor the deployed Event Streams Kafka cluster.

Viewing the preconfigured dashboard

To get an overview of the cluster health, you can view a selection of metrics on the Event Streams Monitoring dashboard.

  1. Log in to your Event Streams UI as an administrator from a supported web browser (see how to determine the login URL for your Event Streams UI).
  2. Click Monitoring in the primary navigation. A dashboard is displayed with overview charts for messages, partitions, and replicas.
  3. Select 1 hour, 1 day, 1 week, or 1 month to view data for different time periods.