Upgrading

Upgrade your Event Streams installation as follows. The Event Streams operator handles the upgrade of your Event Streams instance.

Upgrade paths

You can upgrade Event Streams to the latest 11.3.x version directly from any 11.2.x version by using operator version 3.3.x. The upgrade procedure depends on whether you are upgrading to a major, minor, or patch level version, and what your catalog source is.

On OpenShift, you can upgrade to the latest version by using operator channel v3.3. Review the general upgrade prerequisites before following the instructions to upgrade on OpenShift.

Note: If your operator upgrades are set to automatic, patch level upgrades are completed automatically. This means that the Event Streams operator is upgraded to the latest 3.3.x version when it is available in the catalog, and your Event Streams instance is then also automatically upgraded, unless you set a schedule for the upgrade by pausing the reconciliation.
On other Kubernetes platforms, you must update the Helm repository for any level version update (any digit update: major, minor, or patch), and then upgrade by using the Helm chart. Review the general upgrade prerequisites before following the instructions to upgrade on other Kubernetes platforms.

Prerequisites

The images for Event Streams release 11.3.x are available in the IBM Cloud Container Registry. Ensure you redirect your catalog source to use icr.io/cpopen as described in Implementing ImageContentSourcePolicy to redirect to the IBM Container Registry.
To upgrade successfully, your Event Streams instance must have more than one ZooKeeper node or have persistent storage enabled. If you upgrade an Event Streams instance with a single ZooKeeper node that has ephemeral storage, all messages and all topics will be lost and both ZooKeeper and Kafka pods will move to an error state. To avoid this issue, increase the number of ZooKeeper nodes before upgrading as follows:
```
 apiVersion: eventstreams.ibm.com/v1beta2
 kind: EventStreams
 metadata:
   name: example-pre-upgrade
   namespace: myproject
 spec:
   # ...
   strimziOverrides:
     zookeeper:
       replicas: 3
```
If you installed the Event Streams operator to manage instances of Event Streams in any namespace (one per namespace), then you might need to control when each of these instances is upgraded to the latest version. You can control the updates by pausing the reconciliation of the instance configuration as described in the following sections.
If you are running Event Streams as part of IBM Cloud Pak for Integration, ensure you meet the following requirements:
- Follow the upgrade steps for IBM Cloud Pak for Integration before upgrading Event Streams.
- If you are planning to configure Event Streams with Keycloak, ensure you have IBM Cloud Pak for Integration 2023.4.1 (operator version 7.2.0) or later installed, including the required dependencies.
  
  Note: After upgrading Event Streams to the latest version, if you are changing authentication type from IAM to Keycloak, modify the EventStreams custom resource as described in post-upgrade tasks.

Note: There is no downtime during the Event Streams upgrade. The Kafka pods are rolled one at a time, so a Kafka instance will always be present to serve traffic. However, if the number of brokers you have matches the min.insync.replicas value set for any of your topics, then that topic will be unavailable to write to while the Kafka pods are rolling.

If you have overridden the certificates for any of the Kafka listeners by using the BrokerCertChainAndKey configuration, you must now provide the public Certificate Authority (CA) certificates that were used to sign the overriding certificates.

Ensure you create the secret containing the external CA certificates before you upgrade to avoid Event Streams pods going into a failed state during the upgrade.

Scheduling the upgrade of an instance

In 11.1.x and later, the Event Streams operator handles the upgrade of your Event Streams instance automatically after the operator is upgraded. No additional step is required to change the instance (product) version.

If your operator manages more than one instance of Event Streams, you can control when each instance is upgraded by pausing the reconciliation of the configuration settings for each instance, running the upgrade, and then unpausing the reconciliation when ready to proceed with the upgrade for a selected instance.

Pausing reconciliation by using the CLI

Log in to your Kubernetes cluster as a cluster administrator by setting your kubectl context.
To apply the annotation first to the EventStreams and then to the Kafka custom resource, run the following command, where <type> is either EventStreams or Kafka:
```
kubectl annotate <type> <instance-name> -n <instance-namespace> eventstreams.ibm.com/pause-reconciliation='true'
```
Follow the steps to upgrade on OpenShift.

Unpausing reconciliation by using the CLI

To unpause the reconciliation and continue with the upgrade of an Event Streams instance, run the following command to first remove the annotations from the Kafka custom resource, and then from the EventStreams custom resource, where <type> is either Kafka or EventStreams:

kubectl annotate <type> <instance-name> -n <instance-namespace> eventstreams.ibm.com/pause-reconciliation-

When the annotations are removed, the configuration of your instance is updated, and the upgrade to the latest version of Event Streams completes.

Pausing reconciliation by using the OpenShift web console

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators in the navigation on the left, and click Installed Operators.
From the Project list, select the namespace (project) the instance is installed in.
Locate the operator that manages your Event Streams instance in the namespace. It is called Event Streams in the Name column. Click the Event Streams link in the row.
Select the instance you want to pause and click the YAML tab.

In the YAML for the custom resource, add eventstreams.ibm.com/pause-reconciliation: 'true' to the metadata.annotations field as follows:

apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
metadata:
  name: <instance-name>
  namespace: <instance-namespace>
  annotations:
    eventstreams.ibm.com/pause-reconciliation: 'true'
spec:
  # ...

This annotation also needs to be applied to the corresponding Kafka custom resource. Expand Home in the navigation on the left, click API Explorer, and type Kafka in the Filter by kind... field. Select Kafka.
From the Project list, select the namespace (project) the instance is installed in and click the Instances tab.
Select the instance with the name <instance-name> (the same as the Event Streams instance).

In the YAML for the custom resource, add eventstreams.ibm.com/pause-reconciliation: 'true' to the metadata.annotations field as follows:

apiVersion: eventstreams.ibm.com/v1beta2
kind: Kafka
metadata:
name: <instance-name>
namespace: <instance-namespace>
annotations:
   eventstreams.ibm.com/pause-reconciliation: 'true'

Follow the steps to upgrade on OpenShift.

Unpausing reconciliation by using the OpenShift web console

To unpause the reconciliation and continue with the upgrade of an Event Streams instance, first remove the annotations from the Kafka custom resource, and then from the EventStreams custom resource. When the annotations are removed, the configuration of your instance is updated, and the upgrade to the latest version of Event Streams completes.

Upgrading on the OpenShift Container Platform

Upgrade your Event Streams instance running on the OpenShift Container Platform by using the CLI or web console as follows.

Planning your upgrade

Complete the following steps to plan your upgrade on OpenShift.

Determine which Operator Lifecycle Manager (OLM) channel is used by your existing Subscription. You can check the channel you are subscribed to in the web console (see Update channel section), or by using the CLI as follows (this is the subscription created during installation):
1. Run the following command to check your subscription details:
```
oc get subscription
```
2. Check the CHANNEL column for the channel you are subscribed to, for example, v11.2 in the following snippet:
```
NAME                        PACKAGE                     SOURCE                      CHANNEL
ibm-eventstreams            ibm-eventstreams            ibm-eventstreams-catalog    v3.2
```
If your existing Subscription does not use the v3.3 channel, your upgrade is a change in a minor version. Complete the following steps to upgrade:
1. Ensure the catalog source for new version is available.
2. Change your Subscription to the v3.3 channel by using the CLI or the web console. The channel change will upgrade your operator, and then the operator will upgrade your Event Streams instance automatically.
If your existing Subscription is already on the v3.3 channel, your upgrade is a change to the patch level (third digit) only. Make the catalog source for your new version available to upgrade to the latest level. If you installed by using the IBM Operator Catalog with the latest label, new versions are automatically available. The operator will upgrade your Event Streams instance automatically.

Making new catalog source available

Before you can upgrade to the latest version, the catalog source for the new version must be available on your cluster. Whether you have to take action depends on how you set up version control for your deployment.

Latest versions: If your catalog source is the IBM Operator Catalog, latest versions are always available when published, and you do not have to make new catalog sources available.
Specific versions: If you applied a catalog source for a specific version to control the version of the operator and instances that are installed, you must apply the new catalog source you want to upgrade to.

Upgrading Subscription by using the CLI

If you are using the OpenShift command-line interface (CLI), the oc command, complete the steps in the following sections to upgrade your Event Streams installation.

Ensure the required Event Streams Operator Upgrade Channel is available:

oc get packagemanifest ibm-eventstreams -o=jsonpath='{.status.channels[*].name}'

Change the subscription to move to the required update channel, where vX.Y is the required update channel (for example, v3.3):
```
oc patch subscription -n <namespace> ibm-eventstreams --patch '{"spec":{"channel":"vX.Y"}}' --type=merge
```

All Event Streams pods that need to be updated as part of the upgrade will be gracefully rolled. Where required, ZooKeeper pods will roll one at a time, followed by Kafka brokers rolling one at a time.

Upgrading Subscription by using the web console

If you are using the web console, complete the steps in the following sections to upgrade your Event Streams installation.

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators in the navigation on the left, and click Installed Operators.
From the Project list, select the namespace (project) the instance is installed in.
Locate the operator that manages your Event Streams instance in the namespace. It is called Event Streams in the Name column. Click the Event Streams link in the row.
Click the Subscription tab to display the Subscription details for the Event Streams operator.
Click the version number link in the Update channel section (for example, v3.2). The Change Subscription update channel dialog is displayed, showing the channels that are available to upgrade to.
Select v3.3 and click the Save button on the Change Subscription Update Channel dialog.

Note: The number of containers in each Kafka broker will reduce from 2 to 1 as the TLS-sidecar container will be removed from each broker during the upgrade process.

Upgrading on other Kubernetes platforms by using Helm

If you are running Event Streams on Kubernetes platforms that support the Red Hat Universal Base Images (UBI) containers, you can upgrade Event Streams by using the Helm chart.

Planning your upgrade

Complete the following steps to plan your upgrade on other Kubernetes platforms.

Determine the chart version for your existing deployment:

Change to the namespace where your Event Streams instance is installed:
```
kubectl config set-context --current --namespace=<namespace>
```
Run the following command to check what version is installed:
```
helm list
```

Check the version installed in the CHART column, for example, <chart-name>-3.2.5 in the following snippet:

NAME                      NAMESPACE  REVISION  UPDATED                                 STATUS   CHART                            APP VERSION    
ibm-eventstreams          es         1         2023-11-20 11:49:27.221411789 +0000 UTC deployed ibm-eventstreams-operator-3.2.5  3.2.5

Check the latest chart version that you can upgrade to:
1. Log in to your Kubernetes cluster as a cluster administrator by setting your kubectl context.
2. Add the IBM Helm repository:
```
helm repo add ibm-helm https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm
```
3. Update the Helm repository:
```
helm repo update ibm-helm
```
4. Check the version of the chart you will be upgrading to is the intended version:
```
helm show chart ibm-helm/ibm-eventstreams-operator
```
  Check the version: value in the output, for example: version: 3.3.1
If the chart version for your existing deployment is 3.2.x, your upgrade is a change in a minor version. If your existing chart version is 3.3.x, your upgrade is a change in the patch level version only. Complete the steps in upgrading by using Helm to update your Custom Resource Definitions (CRDs) and operator charts to the latest version. The operator will then upgrade your Event Streams instance automatically.

Upgrading by using Helm

You can upgrade your Event Streams on other Kubernetes platforms by using Helm.

To upgrade Event Streams to the latest version, run the following command:

helm upgrade \
<release-name> ibm-helm/ibm-eventstreams-operator \
-n <namespace> \
--set watchAnyNamespace=<true/false>
--set previousVersion=<previous-version>

Where:

<release-name> is the name you provide to identify your operator.
<namespace> is the name of the namespace where you want to install the operator.
watchAnyNamespace=<true/false> determines whether the operator manages instances of Event Streams in any namespace or only a single namespace (default is false if not specified). For more information, see choosing operator installation mode.
<previous-version> is the version of the Helm chart being upgraded from. For example, if your Helm chart version is 3.2.5, set the field as: --set previousVersion=3.2.5. You can retrieve the version of your existing Helm chart by running the following command:
```
helm list --filter <release-name> -n <namespace> -o json | jq '.[0].app_version'
```

Post-upgrade tasks

Enable collection of producer metrics

In Event Streams version 11.0.0 and later, a Kafka Proxy handles gathering metrics from producing applications. The information is displayed in the Producers dashboard. The proxy is optional and is not enabled by default. To enable metrics gathering and have the information displayed in the dashboard, enable the Kafka Proxy.

Enable metrics for monitoring

To display metrics in the monitoring dashboards of the Event Streams UI:

If you are running Event Streams on the OpenShift Container Platform, ensure you enable the monitoring stack.
If you are running Event Streams on other Kubernetes platforms, you can use any monitoring solution compatible with Prometheus and JMX formats to collect, store, visualize, and set up alerts based on metrics provided by Event Streams.

Migrate to latest Apicurio Registry

Apicurio client libraries versions 2.3.1 and earlier use a date format that is not compatible with Apicurio Registry server versions 2.5.0 or later. Apicurio Registry is only deployed when you update the Event Streams custom resource to use the latest version of Apicurio Registry included with Event Streams.

Migrate your schema registry to use the latest Apicurio Registry as follows:

Ensure all applications connecting to your instance of Event Streams that use the schema registry are using Apicurio client libraries version 2.5.0 or later before migrating.
Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the oc CLI (oc login).
Add the eventstreams.ibm.com/apicurio-registry-version='>=2.4' annotation to your Event Streams custom resource with the following command:
```
oc annotate --namespace <namespace> EventStreams <instance-name> eventstreams.ibm.com/apicurio-registry-version='>=2.4'
```

The Event Streams operator will update your schema registry to use the latest version of Apicurio Registry included with Event Streams.

Configure your instance to use Keycloak

If your existing instance is configured to use IAM and you want to use Keycloak, update your EventStreams custom resource as follows:

Remove the spec.requestIbmServices section.

Set the adminUI authentication type to integrationKeycloak:

# ...
spec:
  # ...
  adminUI:
    authentication:
      - type: integrationKeycloak

Update authentication mechanisms

In versions 11.3.0 and later, each authentication mechanism you want to use for the Event Streams REST endpoints must have a corresponding Kafka listener configured with the same authentication type set.

If you are upgrading an existing installation where you have a specific authentication mechanism set (for example, tls), and at upgrade you do not have a Kafka listener configured with the same authentication type (tls), then the upgrade process will warn you of not having a Kafka listener with the corresponding tls authentication type available.

In previous Event Streams versions, all authentication mechanisms were enabled by default if you did not set any specific mechanism (iam-bearer,tls,scram-sha-512; with iam-bearer only available if you have IBM Cloud Pak foundational services installed on the OpenShift Container Platform). In such cases, ensure you have a Kafka listener set up with the authentication type set to the authentication mechanism you want to use for your REST endpoints.

For example, the following configuration in Event Streams 11.2.5 enabled all authentication mechanisms for the REST endpoint as none were specified. After upgrading to 11.3.x, this configuration means that only SCRAM can be used as the authentication mechanism.

# ...
spec:
  # ...
  adminApi:
    endpoints:
      - name: routes-example
        containterPort: 9080
        type: route

  # ...
  strimziOverrides:
    # ...
    kafka:
      listeners:
        - name: intscram
          type: internal
          port: 9092
          tls: false
          authentication:
            type: scram-sha-512

To enable other authentication mechanisms, for example, TLS, add an additional Kafka listener configuration with tls set as follows:

# ...
spec:
  # ...
  adminApi:
    endpoints:
      - name: routes-example
        containterPort: 9080
        type: route

  # ...
  strimziOverrides:
    # ...
    kafka:
      listeners:
        - name: intscram
          type: internal
          port: 9092
          tls: false
          authentication:
            type: scram-sha-512
        - name: inttls
          type: internal
          port: 9093
          tls: true
          authentication:
            type: tls

Verifying the upgrade

After the upgrade, verify the status of Event Streams by using the CLI or the UI.