Event Streams metrics unavailable

Symptoms

After upgrading to IBM Cloud Pak foundational services version 3.8, the Event Streams UI monitoring page no longer displays the Event Streams metrics and the Event Streams custom dashboards no longer display data.

Causes

Event Streams uses the Prometheus provided with foundational services to scrape and store metrics. The Event Streams UI and the Event Streams custom dashboards are configured to use the data stored in this instance of Prometheus.

IBM Cloud Pak foundational services 3.8 and later does not include Prometheus.

Resolving the problem

To scrape and store metrics, use the Prometheus provided with the Red Hat OpenShift Container Platform.

Reconfigure Event Streams to utilize the OpenShift Container Platform Prometheus instance.

1. Enable OpenShift monitoring for user-defined projects.
2. Create the following podmonitor custom resources:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: <release>-kafka
  namespace: <namespace>
spec:
  podMetricsEndpoints:
  - port: tcp-prometheus
    scheme: http
  selector:
    matchLabels:
      eventstreams.ibm.com/cluster: <release>
      eventstreams.ibm.com/kind: Kafka
      eventstreams.ibm.com/name: <release>-kafka

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: <release>-metrics
  namespace: <namespace>
spec:
  podMetricsEndpoints:
  - port: metrics
    scheme: http
  selector:
    matchLabels:
      eventstreams.ibm.com/name: <release>-ibm-es-metrics

After a period of time (testing showed 20 minutes), the Event Streams custom Kafka dashboard displays metrics.

For information about accessing the Kafka dashboard, see monitoring cluster health using Grafana.

You can use the metrics displayed in this dashboard for monitoring the Event Streams cluster. The dashboard provides the equivalent information that would be displayed in the Event Streams UI.