Attention: This version of Event Streams has reached End of Support. For more information about supported versions, see the support matrix.

Monitoring Kafka consumer group lag

You can monitor the consumer lag for Kafka clients connecting to Event Streams. This information is available through both the Event Streams UI and CLI.

Consumer lag

Each partition will have a consumer within a consumer group with information relating to its consumption as follows:

  • Client ID and Consumer ID: each partition will have exactly one consumer in the consumer group, identifiable by a Client ID and Consumer ID.
  • Current offset: the last committed offset of the consumer.
  • Log end offset: the highest offset of the partition.
  • Offset lag: the difference between the current consumer offset and the highest offset, which shows how far behind the consumer is.

Using the Event Streams UI

To access the consumer groups side panel in the Event Streams UI, do the following:

  1. Log in to the Event Streams UI from a supported web browser (see how to determine the login URL for your Event Streams UI).
  2. Click Topics in the primary navigation.
  3. Locate your topic using the Name column and click the row for the topic.
  4. Click the Consumer groups tab.
  5. The Consumer groups dashboard will display all consumer groups for the selected topic. Click the consumer group of interest from the Consumer group ID column. The consumer group side panel should now be displayed on screen.

This side panel will display a table containing consumer group information for each partition of the topic.

Using the Event Streams CLI

To access information about a consumer group in the Event Streams CLI, do the following:

  1. Log in to your cluster as an administrator by using the IBM Cloud Pak CLI:
    cloudctl login -a https://<cluster_address>:<cluster_router_https_port>
  2. Run the following command to initialize the Event Streams CLI on the cluster:
    cloudctl es init
  3. To list all consumer groups on a cluster, run:
    cloudctl es groups
  4. To list information about a consumer group, run:
    cloudctl es group --group <consumer-group-id>
    where <consumer-group-id> is the name of the consumer group of interest.

The CLI will print a table containing consumer group information for each partition of the topic.

The following example shows the information the command returns for a consumer group called my-consumer:

$ cloudctl es group --group my-consumer
Details for consumer group my-consumer
Group ID            State
my-consumer         Stable

Topic      Partition   Current offset   End offset   Offset lag   Client        Consumer        Host
my-topic   2           999              1001         2            some-client   some-consumer   some-host
my-topic   0           1000             1001         1            some-client   some-consumer   some-host
my-topic   1           1000             1000         0            some-client   some-consumer   some-host

To view other Kafka-related metrics, consider configuring a Kafka Exporter.

For information on how to monitor the general health of a particular topic, see monitoring topic health.