Upgrading and migrating

Upgrade your Event Streams operator and operand instances as follows.

Upgrade paths

Upgrade to Event Streams 11.0.x by following these supported upgrade paths.

Upgrade paths for EUS releases

You can upgrade to the Event Streams CD release 11.0.4 from the latest version of the Extended Update Support (EUS) update channel. The latest operator and operand version of the EUS release ensures that you have the updates to enable upgrading to the CD release.

Before you upgrade to the CD release, upgrade your Event Streams EUS release to the following versions:

Earlier operator versions must be upgraded to 2.2.6.
Earlier operand versions must be upgraded to 10.2.1-eus.

Upgrade paths for CD releases

The following upgrade paths are available for Continuous Delivery (CD) releases (2.x operators and 10.x operands and later, except for 2.2.x operators and 10.2.x operands):

You can upgrade the Event Streams operator to the 3.0.5 version directly from versions 3.0.x, 2.5.x, and 2.4.x. Earlier operator versions must be upgraded to 2.4.0 before upgrading to 3.0.5.
You can upgrade the Event Streams operand to the 11.0.4 version directly from versions 11.0.x, 10.5.x, and 10.4.x. Earlier operand versions must be upgraded to 10.4.0 before upgrading to 11.0.4.

Prerequisites

Ensure you have followed the upgrade steps for IBM Cloud Pak for Integration before upgrading Event Streams.

Note: If you upgrade your Red Hat OpenShift version before upgrading your Event Streams operator version, the Event Streams operand might display a Failed status temporarily. This will be resolved when you upgrade your operator and then the operand version, the next steps in the upgrade process.
The images for Event Streams release 11.0.x are available in the IBM Cloud Container Registry. Ensure you redirect your catalog source to use icr.io/cpopen as described in Implementing ImageContentSourcePolicy to redirect to the IBM Container Registry.
To upgrade successfully, your Event Streams instance must have more than one ZooKeeper node or have persistent storage enabled. If you upgrade an Event Streams instance with a single ZooKeeper node that has ephemeral storage, all messages and all topics will be lost and both ZooKeeper and Kafka pods will move to an error state. To avoid this issue, increase the number of ZooKeeper nodes before upgrading as follows:
```
 apiVersion: eventstreams.ibm.com/v1beta1
 kind: EventStreams
 metadata:
   name: example-pre-upgrade
   namespace: myproject
 spec:
   strimziOverrides:
     zookeeper:
       replicas: 3
```
If you have an Event Streams 11.0.3 or earlier installation, and you previously added your own Kafka or Zookeeper metrics rules, then ensure you record these elsewhere to be added to the metrics-config ConfigMap after upgrading.

Important: If you have configured custom certificates on any of the listeners in your Event Streams custom resource, there will be an outage when you upgrade the operand to any 11.0.x version until you reconfigure your listeners with your custom certificates. This is because the upgrade involves a migration of the listener format, which removes any reference to the custom certificates. To ensure you are prepared to quickly deal with this outage after a successful upgrade, review how you can reconfigure your custom certificates.

Back up your instance configuration

Before you upgrade, create a backup of your Event Streams instance configuration by completing the following steps.

If you are using Red Hat OpenShift command-line interface (CLI):

Run the following command:

oc get eventstreams <instance-name> -n <namespace> -o yaml > <filename>.yaml

If you are using the OpenShift Container Platform web console, complete the following steps:

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators in the navigation on the left, and click Installed Operators.
From the Project list, select the namespace (project) the instance is installed in.
Locate the row for Event Streams operator and click the Event Streams link in the Provided APIs column.
Locate the row for Event Streams instance and click the link in the Name column.
Click the YAML tab to display the configuration details for the Event Streams instance.
Copy the definition of your Event Streams instance and save it as a YAML file.

Upgrading from EUS releases

Before upgrading your EUS release, ensure you:

Review the guidance in prerequisites.
Create a back up your instance configuration.
Upgrade your IBM Cloud Pak foundational services from EUS version 3.6.x to the latest CD version, and then clean up the monitoring resources.

Uninstall the Event Streams operator 2.2.6

Uninstall the existing version (2.2.6) of the Event Streams operator by using OpenShift Container Platform web console.

Important: The Event Streams instance continues to run and it’s configuration is not deleted. However, Event Streams instance is not managed by an operator in this state.

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators and click Installed Operators.
In the Project dropdown, select the required namespace. For cluster-wide operators, select the openshift-operators project.
Click More options next to the Event Streams operator to be deleted to open the actions menu.
Click the Uninstall Operator menu option to open the confirmation panel.
Check the namespace and operator name, then click Remove to uninstall the operator.

Install the Event Streams operator 3.0.5

After uninstalling the 2.2.6 operator, install the Event Streams operator 3.0.5 from v3.0 update channel by using OpenShift Container Platform web console. The installation can take a few minutes to complete. All Event Streams pods that need to be updated as part of the upgrade will be gracefully rolled.

Log in to the OpenShift Container Platform web console using your login credentials.
Expand the Operators dropdown and select OperatorHub to open the OperatorHub dashboard.
Select the project that you want to deploy the Event Streams instance in.
In the All Items search box, enter Event Streams to locate the operator title.
Click the Event Streams tile to open the installation side panel.
Click the Install button to open the Create Operator Subscription dashboard.
Select the v3.0 update channel and the installation mode that suits your requirements. If the installation mode is A specific namespace on the cluster, select the target namespace.
Click Install to begin the installation and wait for it to complete.

Important: The Entity operator might display a CrashLoopBackOff error. In addition, some Event Streams pods might display errors temporarily when the operator version has been updated. You can ignore these errors and continue with upgrading the Event Streams operand version, which will resolve these errors.

Note: The number of containers in each Kafka broker will reduce from 2 to 1 as the TLS-sidecar container will be removed from each broker during the upgrade process.

Upgrade the Event Streams operand (instance)

Upgrade the Event Streams instance from the existing EUS version (10.2.1-eus) to the required version 11.0.4. All Event Streams pods will gracefully roll again.

If you are using Red Hat OpenShift command-line interface (CLI), run the following command:

oc patch eventstreams -n <namespace> <name-of-the-es-instance> --patch '{"spec":{"version":"11.0.4"}}' --type=merge

If you are using the OpenShift Container Platform web console, complete the following steps:

Click Installed Operators from the navigation on the left to view the list of installed operators, including the upgraded Event Streams operator.
Select the Event Streams operator from the list of Installed Operators.
Click the Event Streams tab. This lists the Event Streams operands.
Find your instance in the Name column and click the link for the instance.
Click the YAML tab. The Event Streams instance custom resource is shown.
In the YAML, change the spec.version field to the required version 11.0.4.
Click the Save button.

Upgrading from CD releases

Ensure you have reviewed the guidance in prerequisites and back up your instance configuration sections and then follow all the steps that are outlined in this.

Upgrade the Event Streams operator

Update the subscription for the Event Streams operator from the existing CD channel (v2.4 or v2.5) to the required update channel v3.0. All Event Streams pods that need to be updated as part of the upgrade will be gracefully rolled. Where required, ZooKeeper pods roll one at a time, followed by Kafka brokers rolling one at a time.

If you are using Red Hat OpenShift command-line interface (CLI), run the following commands:

Ensure the required Event Streams Operator Upgrade Channel is available:

oc get packagemanifest ibm-eventstreams -o=jsonpath='{.status.channels[*].name}'

Change the subscription to move to the required update channel v3.0:

oc patch subscription -n <namespace> ibm-eventstreams --patch '{"spec":{"channel":"v3.0"}}' --type=merge

If you are using OpenShift Container Platform web console, complete the following steps:

Log in to the OpenShift Container Platform web console using your login credentials.
Expand Operators in the navigation on the left, and click Installed Operators.
From the Project list, select the namespace (project) the instance is installed in.
Locate the operator that manages your Event Streams instance in the namespace. It is called Event Streams in the Name column. Click the Event Streams link in the row.
Click the Subscription tab to display the Subscription details for the Event Streams operator.
Click the version number link in the Update channel section (for example, v2.5). The Change Subscription update channel dialog is displayed, showing the channels that are available to upgrade to.
Select v3.0 and click the Save button on the Change Subscription Update Channel dialog.

Note: The number of containers in each Kafka broker will reduce from 2 to 1 as the TLS-sidecar container will be removed from each broker during the upgrade process.

Upgrade the Event Streams operand (instance)

Upgrade the Event Streams instance from the existing CD version (10.4.x or 10.5.x) to the required version 11.0.4. All Event Streams pods will gracefully roll again.

If you are using Red Hat OpenShift command-line interface (CLI), run the following command:

oc patch eventstreams -n <namespace> <name-of-the-es-instance> --patch '{"spec":{"version":"11.0.4"}}' --type=merge

If you are using the OpenShift Container Platform web console, complete the following steps:

Click Installed Operators from the navigation on the left to view the list of installed operators, including the upgraded Event Streams operator.
Select the Event Streams operator from the list of Installed Operators.
Click the Event Streams tab. This lists the Event Streams operands.
Find your instance in the Name column and click the link for the instance.
Click the YAML tab. The Event Streams instance custom resource is shown.
In the YAML, change the spec.version field to the required version 11.0.4.
Click the Save button.

If you are using the IBM Cloud Pak for Integration Platform UI, complete the following steps:

Log in to the IBM Cloud Pak for Integration Platform UI.
Click the Navigation menu in the upper left.
Expand Administration and click Integration instances. If an update is available for a runtime, the Information icon displays next to the runtime’s current Version number.
Click the More options in the row for the Event Streams instance, and then click Change version.
Select 11.0.4 from the Select a new channel or version list.
Click Change version to save your selections and start the upgrade. In the runtimes table, the Status column for the runtime displays the Upgrading message. The upgrade is complete when the Status is Ready and the Version displays the new version number.

Verifying the upgrade

Wait for all Event Streams pods to complete the upgrade process. This is indicated by the Running state.
Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the oc CLI (oc login).
To retrieve a list of Event Streams instances, run the following command:
oc get eventstreams -n <namespace>
For the instance of Event Streams that you upgraded, check that the status returned by the following command is Ready.
oc get eventstreams -n <namespace> <name-of-the-es-instance> -o jsonpath="'{.status.phase}'"

Post-upgrade tasks

Reconfigure listeners with custom certificates

To enable your clients to connect with the upgraded Event Streams that uses custom certificates, add the certificate configuration again in the new format as follows:

      - name: external
        type: route
        port: 9092
        authentication:
          type: scram-sha-512
        tls: true
        configuration:
          brokerCertChainAndKey:
            certificate: my-listener-certificate.crt
            key: my-listener-key.key
            secretName: my-secret

Enable collection of producer metrics

In Event Streams version 11.0.0 and later, a Kafka Proxy handles gathering metrics from producing applications. The information is displayed in the Producers dashboard. The proxy is optional and is not enabled by default. To enable metrics gathering and have the information that is displayed in the dashboard, enable the Kafka Proxy.

Update metrics rules

If you previously added your own Kafka or Zookeeper metrics rules, then ensure you add these rules again to the metrics-config ConfigMap. The rules in the following ConfigMap are the default rules. Add your own custom rules to data.kafka-metrics-config.yaml.rules: or data.zookeeper-metrics-config.yaml.rules: after the default rules.

If you do not want to export metrics then delete all of the rules in the metrics-config ConfigMap.

kind: ConfigMap
apiVersion: v1
metadata:
  name: metrics-config
data:
  kafka-metrics-config.yaml: |
    lowercaseOutputName: true
    rules:
    - attrNameSnakeCase: false
      name: kafka_controller_$1_$2_$3
      pattern: kafka.controller<type=(\w+), name=(\w+)><>(Count|Value|Mean)
    - attrNameSnakeCase: false
      name: kafka_server_BrokerTopicMetrics_$1_$2
      pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec)><>(Count)
    - attrNameSnakeCase: false
      name: kafka_server_BrokerTopicMetrics_$1__alltopics_$2
      pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec)><>(OneMinuteRate)
    - attrNameSnakeCase: false
      name: kafka_server_ReplicaManager_$1_$2
      pattern: kafka.server<type=ReplicaManager, name=(\w+)><>(Value)
  zookeeper-metrics-config.yaml: |
    lowercaseOutputName: true
    rules: []

Deprecation warnings for metrics

The metrics configuration option for the EventStreams custom resource has been deprecated. If you receive deprecation warnings about spec.kafka.metrics: {} or spec.zookeeper.metrics:{} after upgrading, then remove the metrics: {} line from the EventStreams custom resource.

Enable metrics for monitoring

To display metrics in the monitoring dashboards of the Event Streams UI, ensure you enable the OpenShift Container Platform monitoring stack.