Monitoring the health of your Kafka cluster helps to verify that your operations are running smoothly. The Event Streams UI includes a preconfigured dashboard that monitors Kafka data.
Event Streams also provides a number of ways to export metrics from your Kafka brokers to external monitoring and logging applications. These metrics are useful indicators of the health of the cluster, and can provide warnings of potential problems. You can use any monitoring solution that is compatible with Prometheus and JMX formats to collect, store, visualize, and set up alerts based on metrics provided by Event Streams. The following sections provide an overview of the available options.
For information about the health of your topics, check the producer activity dashboard.
You can use Event Streams to collect JMX metrics from Kafka brokers, ZooKeeper nodes, and Kafka Connect nodes, and export them to Prometheus.
For an example of how to configure the JMX exporter, see configuring the JMX Exporter.
You can use Event Streams to export Kafka metrics to Prometheus. These metrics are otherwise only accessible through the Kafka command-line tools. This allows topic metrics such as consumer group lag to be collected.
For an example of how to configure a Kafka Exporter, see configuring the Kafka Exporter.
You can use the Kibana service that is provided by the OpenShift Container Platform cluster logging, and use the example Kibana dashboards to monitor for specific errors in the logs and set up alerts for when a number of errors occur over a period of time in your Event Streams instance.
To install the Event Streams Kibana dashboards, follow these steps:
- Ensure you have cluster logging installed.
Download the JSON file that includes the example Kibana dashboards for Event Streams from GitHub.
Navigate to the Kibana homepage on your cluster.
- Click Management in the navigation on the left.
- Click Index patterns.
- Click Create index pattern.
app*in the Index pattern field, and click Next step.
@timestampfrom the Time Filter field name list, and click Create index pattern.
- Click Saved Objects.
- Click the Import icon and navigate to the JSON file you downloaded earlier that includes the example Kibana dashboards for Event Streams.
- If an
Index Pattern Conflictswarning is displayed, select the
app*index pattern from the New index pattern list for each conflict, then click Confirm all changes.
- Click Dashboard in the navigation on the left to view the downloaded dashboards.
You can also use Kibana on other Kubernetes platforms, where it might also be included, or you can install Kibana externally. You can use the example Kibana dashboards to monitor for specific errors in the logs and set up alerts for when a number of errors occur over a period of time in your Event Streams instance.
Instana is an observability tool that can be used to monitor your Event Streams deployment.
Instana also offers Kafka-centric monitoring that can provide useful insights into the performance and the health of your Kafka cluster.
For information about installing and configuring an Instana host agent on the Red Hat OpenShift Container Platform, see the Instana documentation for OpenShift. To use Instana on other Kubernetes platforms, see the Instana documentation for Kubernetes.
After installing, Instana can monitor all aspects of an Event Streams instance with no extra configuration required.
Note: You might receive the following error message in the Instana dashboards when you check monitoring metrics for the Event Streams UI container:
Monitoring issue: nodejs_collector_not_installed
The @instana/collector package is not installed in this Node.js application, or the @instana/collector package cannot announce itself to the host agent, for example due to networking issues.
If you require monitoring of the Event Streams UI, you can enable Instana to monitor the UI by setting the following in the
EventStreams custom resource:
- name: INSTANA_AGENT_HOST
Other Monitoring Tools
You can also use external monitoring tools to monitor the deployed Event Streams Kafka cluster.
Viewing the preconfigured dashboard
To get an overview of the cluster health, you can view a selection of metrics on the Event Streams Monitoring dashboard.
- Log in to your Event Streams UI as an administrator from a supported web browser (see how to determine the login URL for your Event Streams UI).
- Click Monitoring in the primary navigation. A dashboard is displayed with overview charts for messages, partitions, and replicas.
- Select 1 hour, 1 day, 1 week, or 1 month to view data for different time periods.