Attention: This version of Event Streams has reached End of Support. For more information about supported versions, see the support matrix.

Monitoring Kafka cluster health

Monitoring the health of your Kafka cluster ensures your operations run smoothly. Event Streams collects metrics from all of the Kafka brokers and exports them to a Prometheus-based monitoring platform. The metrics are useful indicators of the health of the cluster, and can provide warnings of potential problems.

You can use the metrics as follows:

  • View a selection of metrics on a configured dashboard in the Event Streams UI.

  • Create dashboards in the Grafana service that is provided in IBM Cloud Private. You can download example Grafana dashboards for Event Streams from GitHub.

    For more information about the monitoring capabilities provided in IBM Cloud Private, including Grafana, see the IBM Cloud Private documentation.

    To install the configured Grafana dashboards, follow these steps:

    1. Download the dashboards you would like to install from Github.
    2. Log in to your IBM Cloud Private cluster management console from a supported web browser by using the URL https://<Cluster Master Host>:<Cluster Master API Port>. The master host and port for your cluster are set during the installation of IBM Cloud Private. For more information, see the IBM Cloud Private documentation.
    3. Navigate to the IBM Cloud Private console homepage.
    4. Click the hamburger icon on the top left.
    5. Expand Platform.
    6. Click the Monitoring to navigate you to the Grafana homepage.
    7. On the Grafana homepage, click on the Home icon on the top left to bring down a view of all the pre-installed dashboards.
    8. Click on the Import Dashboards and either paste the JSON of the dashboard you want to install or import the dashboard’s JSON file that was downloaded in step 1.
    9. Navigate to the Grafana homepage again and click on the Home icon again then find the Dashboard you have installed to view it.

    Ensure you select your namespace, release name, and other filters at the top of the dashboard to view the required information.

  • Create alerts so that metrics that meet predefined criteria are used to send notifications to emails, Slack, PagerDuty, and so on. For an example of how to use the metrics to trigger alert notifications, see how you can set up notifications to Slack.

  • Create dashboards in the Kibana service that is provided in IBM Cloud Private. You can download example Kibana dashboards for Event Streams from GitHub to monitor for specific errors in the logs and set up alerts for when a number of errors over a period of time in your Event Streams instance.

    For more information about the logging capabilities provided in IBM Cloud Private, including Kibana, see the IBM Cloud Private documentation.

    To download the preconfigured Kibana Dashboards, follow these steps:

    1. Download Event Streams Kibana Dashboard.json from GitHub
    2. Log in to your IBM Cloud Private cluster management console from a supported web browser by using the URL https://<Cluster Master Host>:<Cluster Master API Port>. The master host and port for your cluster are set during the installation of IBM Cloud Private. For more information, see the IBM Cloud Private documentation.
    3. Navigate to the IBM Cloud Private console homepage.
    4. Click the hamburger icon on the top left.
    5. Expand the Platform.
    6. Click the Logging to the Kibana homepage.
    7. Click the Management on the left.
    8. Click on the Saved Objects.
    9. Click the Import icon and navigate the Event Streams Kibana Dashboard.json that you have downloaded.
    10. Click on the Dashboard tab on the left hand side menu and you should see the downloaded dashboards.

You can also use external monitoring tools to monitor the deployed Event Streams Kafka cluster.

For information about the health of your topics, check the producer activity dashboard.

Important: By default, the metrics data used to provide monitoring information is only stored for a day. Modify the time period for metric retention to be able to view monitoring data for longer time periods, such as 1 week or 1 month.

Viewing the preconfigured dashboard

To get an overview of the cluster health, you can view a selection of metrics on the Event Streams Monitor dashboard.

  1. Log in to Event Streams as an administrator
  2. Click the Monitor tab. A dashboard is displayed with overview charts for messages, partitions, and replicas.
  3. Click a chart to drill down into more detail.
  4. Click 1 hour, 1 day, 1 week, or 1 month to view data for different time periods.