Upgrading and migrating

Upgrade your IBM Event Streams operator and operand instances as follows.

Upgrade paths

You must first upgrade the Event Streams operator, and then upgrade your Event Streams instance (operand version).

Upgrade paths for CD releases

The following upgrade paths are available for Continuous Delivery (CD) releases (2.x operators and 10.x operands and later, except for 2.2.x operators and 10.2.x operands):

You can upgrade the Event Streams operator to the latest 3.0.5 version directly from versions 3.0.x, 2.5.x, and 2.4.x. If you have an earlier operator version than 2.4.0, you must first upgrade it to 2.4.0 before upgrading to 3.0.x.
You can upgrade the Event Streams operand to the latest 11.0.4 version directly from versions 11.0.x, 10.5.x, and 10.4.x. If you have an earlier operand version than 10.4.0, you must first upgrade it to 10.4.0 before upgrading to 11.0.x.

Upgrade paths for EUS releases

You can also upgrade to the latest Event Streams CD release from the Event Streams Extended Update Support (EUS) release (2.2.x operators and 10.2.x operands) as follows:

Upgrade your Event Streams EUS release by upgrading to the latest EUS operator (2.2.x). The latest operator revision for the EUS release ensures you have the latest updates and fixes applied, including updates to enable upgrading to the latest CD release.
Upgrade your IBM Cloud Pak foundational services from EUS version 3.6.x to the latest CD version.
After successfully upgrading to the latest CD version of foundational services, ensure you clean up the monitoring resources to avoid errors.
Upgrade your Event Streams version to the latest CD release by following the instructions on this page starting with the prerequisites (operator version 3.0.5 and operand version 11.0.4).

Prerequisites

Ensure you have followed the upgrade steps for IBM Cloud Pak for Integration before upgrading Event Streams.

Note: If you upgrade your OpenShift version before upgrading your Event Streams operator version, the Event Streams operand might display a Failed status temporarily. This will be resolved when you upgrade your operator and then the operand version, the next steps in the upgrade process.
The images for Event Streams release 11.0.x are available in the IBM Cloud Container Registry. Ensure you redirect your catalog source to use icr.io/cpopen as described in Implementing ImageContentSourcePolicy to redirect to the IBM Container Registry.
To upgrade successfully, your Event Streams instance must have more than one ZooKeeper node or have persistent storage enabled. If you upgrade an Event Streams instance with a single ZooKeeper node that has ephemeral storage, all messages and all topics will be lost and both ZooKeeper and Kafka pods will move to an error state. To avoid this issue, increase the number of ZooKeeper nodes before upgrading as follows:
```
 apiVersion: eventstreams.ibm.com/v1beta1
 kind: EventStreams
 metadata:
   name: example-pre-upgrade
   namespace: myproject
 spec:
   strimziOverrides:
     zookeeper:
       replicas: 3
```
If you have an Event Streams 11.0.3 or earlier installation, and you previously added your own Kafka or Zookeeper metrics rules, then ensure you record these elsewhere to be added to the metrics-config ConfigMap after upgrading.

Important: If you have configured custom certificates on any of the listeners in your Event Streams custom resource, there will be an outage when you upgrade the operand to any 11.0.x version until you reconfigure your listeners with your custom certificates. This is because the upgrade involves a migration of the listener format, which removes any reference to the custom certificates. To ensure you are prepared to quickly deal with this outage after a successful upgrade, review how you can reconfigure your custom certificates.

Upgrading by using the UI

The upgrade process requires the upgrade of the Event Streams operator, and then the upgrade of your Event Streams instances. If you are using the OpenShift Container Platform web console, complete the steps in the following sections to upgrade your Event Streams installation.

Upgrade the Event Streams operator

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators in the navigation on the left, and click Installed Operators.
From the Project list, select the namespace (project) the instance is installed in.
Locate the operator that manages your Event Streams instance in the namespace. It is called IBM Event Streams in the Name column. Click the IBM Event Streams link in the row.
Click the Subscription tab to display the Subscription details for the Event Streams operator.
Click the version number link in the Update channel section (for example, v2.5). The Change Subscription update channel dialog is displayed, showing the channels that are available to upgrade to.
Select v3.0 and click the Save button on the Change Subscription Update Channel dialog.

All Event Streams pods that need to be updated as part of the upgrade will be gracefully rolled. Where required, ZooKeeper pods will roll one at a time, followed by Kafka brokers rolling one at a time.

Important: The Entity operator might display a CrashLoopBackOff error. In addition, some Event Streams pods might display errors temporarily when the operator version has been updated. You can ignore these errors and continue with upgrading the Event Streams operand version, which will resolve these errors.

Note: The number of containers in each Kafka broker will reduce from 2 to 1 as the TLS-sidecar container will be removed from each broker during the upgrade process.

Upgrade the Event Streams operand (instance)

Click Installed Operators from the navigation on the left to view the list of installed operators, including the upgraded IBM Event Streams operator.
Select the IBM Event Streams operator from the list of Installed Operators.
Click the Event Streams tab. This lists the Event Streams operands.
Find your instance in the Name column and click the link for the instance.
Click the YAML tab. The Event Streams instance custom resource is shown.
In the YAML, change the spec.version field to the required version, for example, 11.0.4.
Click the Save button.

All Event Streams pods will gracefully roll again.

Alternatively, you can also upgrade your Event Streams instance in the IBM Cloud Pak for Integration Platform UI as follows:

Log in to the IBM Cloud Pak for Integration Platform UI.
Click the Navigation Menu in the top left.
Expand Administration and click Integration instances. If an update is available for a runtime, the Information icon displays next to the runtime’s current Version number.
Click the More options in the row for the Event Streams instance, and then click Change version.
Select 11.0.4 from the Select a new channel or version list.
Click Change version to save your selections and start the upgrade. In the runtimes table, the Status column for the runtime displays the Upgrading message. The upgrade is complete when the Status is Ready and the Version displays the new version number.

Upgrading by using the CLI

The upgrade process requires the upgrade of the Event Streams operator, and then the upgrade of your Event Streams instances. If you are using the OpenShift command-line interface (CLI), the oc command, complete the steps in the following sections to upgrade your Event Streams installation.

Upgrade the Event Streams operator

Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the oc CLI (oc login).
Ensure the required IBM Event Streams Operator Upgrade Channel is available:

oc get packagemanifest ibm-eventstreams -o=jsonpath='{.status.channels[*].name}'
Change the subscription to move to the required update channel, where vX.Y is the required update channel (for example, v3.0):

oc patch subscription -n <namespace> ibm-eventstreams --patch '{"spec":{"channel":"vX.Y"}}' --type=merge

Note: The number of containers in each Kafka broker will reduce from 2 to 1 as the TLS-sidecar container will be removed from each broker during the upgrade process.

Upgrade the Event Streams operand (instance)

Upgrade the Event Streams instance to move to the required version, where X.Y.Z is the required version, for example, 11.0.4.

oc patch eventstreams -n <namespace> <name-of-the-es-instance> --patch '{"spec":{"version":"X.Y.Z"}}' --type=merge

All Event Streams pods will gracefully roll again.

Verifying the upgrade

Wait for all Event Streams pods to complete the upgrade process. This is indicated by the Running state.
Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the oc CLI (oc login).
To retrieve a list of Event Streams instances, run the following command:
oc get eventstreams -n <namespace>
For the instance of Event Streams that you upgraded, check that the status returned by the following command is Ready.
oc get eventstreams -n <namespace> <name-of-the-es-instance> -o jsonpath="'{.status.phase}'"

Post-upgrade tasks

Reconfigure listeners with custom certificates

To enable your clients to connect with the upgraded Event Streams that uses custom certificates, add the certificate configuration again in the new format as follows:

      - name: external
        type: route
        port: 9092
        authentication:
          type: scram-sha-512
        tls: true
        configuration:
          brokerCertChainAndKey:
            certificate: my-listener-certificate.crt
            key: my-listener-key.key
            secretName: my-secret

Enable collection of producer metrics

In IBM Event Streams version 11.0.0 and later, a Kafka Proxy handles gathering metrics from producing applications. The information is displayed in the Producers dashboard. The proxy is optional and is not enabled by default. To enable metrics gathering and have the information displayed in the dashboard, enable the Kafka Proxy.

Update metrics rules

If you previously added your own Kafka or Zookeeper metrics rules, then ensure you add these rules again to the metrics-config ConfigMap. The rules in the following ConfigMap are the default rules. Add your own custom rules to data.kafka-metrics-config.yaml.rules: or data.zookeeper-metrics-config.yaml.rules: after the default rules.

If you do not want to export metrics then delete all of the rules in the metrics-config ConfigMap.

kind: ConfigMap
apiVersion: v1
metadata:
  name: metrics-config
data:
  kafka-metrics-config.yaml: |
    lowercaseOutputName: true
    rules:
    - attrNameSnakeCase: false
      name: kafka_controller_$1_$2_$3
      pattern: kafka.controller<type=(\w+), name=(\w+)><>(Count|Value|Mean)
    - attrNameSnakeCase: false
      name: kafka_server_BrokerTopicMetrics_$1_$2
      pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec)><>(Count)
    - attrNameSnakeCase: false
      name: kafka_server_BrokerTopicMetrics_$1__alltopics_$2
      pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec)><>(OneMinuteRate)
    - attrNameSnakeCase: false
      name: kafka_server_ReplicaManager_$1_$2
      pattern: kafka.server<type=ReplicaManager, name=(\w+)><>(Value)
  zookeeper-metrics-config.yaml: |
    lowercaseOutputName: true
    rules: []

Deprecation warnings for metrics

The metrics configuration option for the EventStreams custom resource has been deprecated. If you receive deprecation warnings about spec.kafka.metrics: {} or spec.zookeeper.metrics:{} after upgrading, then remove the metrics: {} line from the EventStreams custom resource.

Enable metrics for monitoring

To display metrics in the monitoring dashboards of the Event Streams UI, ensure you enable the OpenShift Container Platform monitoring stack.