The following sections provide instructions about installing Event Streams on the Red Hat OpenShift Container Platform. The instructions are based on using the OpenShift Container Platform web console and oc
command-line utility.
Event Streams can also be installed as part of IBM Cloud Pak for Integration.
Overview
Event Streams is an operator-based release and uses custom resources to define your Event Streams configurations. The Event Streams operator uses the custom resources to deploy and manage the entire lifecycle of your Event Streams instances. Custom resources are presented as YAML configuration documents that define instances of the EventStreams
custom resource type.
Installing Event Streams has two phases:
- Install the Event Streams operator: this will deploy the operator that will install and manage your Event Streams instances.
- Install one or more instances of Event Streams by using the operator.
Before you begin
- Ensure you have set up your environment according to the prerequisites, including setting up your OpenShift Container Platform.
- If you want to authenticate with Keycloak, ensure you have IBM Cloud Pak for Integration 2023.4.1 (operator version 7.2.0) or later installed, including the required dependencies.
- Ensure you have planned for your installation, such as preparing for persistent storage, considering security options, and considering adding resilience through multiple availability zones.
- Obtain the connection details for your OpenShift Container Platform cluster from your administrator.
-
The Event Streams UI includes dashboards for monitoring Kafka health and topic health. To provide metrics for these dashboards, ensure you enable the OpenShift Container Platform monitoring stack as described in the IBM Cloud Pak for Integration documentation.
In addition, to provide metrics about topic health, enable the Kafka Proxy.
Create a project (namespace)
Create a namespace into which the Event Streams instance will be installed by creating a project. When you create a project, a namespace with the same name is also created.
Ensure you use a namespace that is dedicated to a single instance of Event Streams. This is required because Event Streams uses network security policies to restrict network connections between its internal components. A single namespace per instance also allows for finer control of user accesses.
Important: Do not use any of the default or system namespaces to install an instance of Event Streams (some examples of these are: default
, kube-system
, kube-public
, and openshift-operators
).
Creating a project by using the web console
- Log in to the OpenShift Container Platform web console using your login credentials.
- Expand the Home dropdown and select Projects to open the Projects panel.
- Click Create Project.
- Enter a new project name in the Name field, and optionally, a display name in the Display Name field, and a description in the Description field.
- Click Create.
Creating a project by using the CLI
- Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the
oc
CLI (oc login
). -
Run the following command to create a new project:
oc new-project <project_name> --description="<description>" --display-name="<display_name>"
Where
description
anddisplay-name
are optional flags to set a description and custom descriptive name for your project. -
Ensure you are using the project you created by selecting it as follows:
oc project <new-project-name>
The following message is displayed if successful:
Now using project "<new-project-name>" on server "https://<OpenShift-host>:6443".
Choose the operator installation mode
Before installing the Event Streams operator, decide if you want the operator to:
-
Manage instances of Event Streams in any namespace.
To use this option, select
All namespaces on the cluster (default)
later. The operator will be deployed into the system namespaceopenshift-operators
, and will be able to manage instances of Event Streams in any namespace. -
Only manage instances of Event Streams in a single namespace.
To use this option, select
A specific namespace on the cluster
later. The operator will be deployed into the specified namespace, and will not be able to manage instances of Event Streams in any other namespace.Important: When multiple Event Streams operators are installed on the same cluster, all the operators share the same custom resource definitions (CRDs). Do not install a previous version of the Event Streams operator if a later version is already installed on the same cluster. Running different versions of Event Streams in different namespaces on the same cluster is not supported, except during upgrade where it is necessary to upgrade each namespace sequentially.
Creating an image pull secret
Before installing an Event Streams instance, create an image pull secret called ibm-entitlement-key
in the namespace where you want to create an instance of Event Streams. The secret enables container images to be pulled from the registry.
- Obtain an entitlement key from the IBM Container software library.
-
Create the secret in the namespace that will be used to deploy an instance of Event Streams as follows.
Name the secret
ibm-entitlement-key
, usecp
as the username, your entitlement key as the password, andcp.icr.io
as the docker server:oc create secret docker-registry ibm-entitlement-key --docker-username=cp --docker-password="<your-entitlement-key>" --docker-server="cp.icr.io" -n <target-namespace>
Note: If you do not create the required secret, pods will fail to start with ImagePullBackOff
errors. In this case, ensure the secret is created and allow the pod to restart.
Creating the catalog sources
Before you can install the required IBM operators, make them available for installation by adding the catalog sources to your cluster. Two options are available for creating your Event Streams catalog sources:
-
Specify the Event Streams version that you want available in your cluster by downloading the Container Application Software for Enterprises (CASE) files for that version. This option is suitable for production deployments where you want to control what versions are available and when upgrades are applied. For more information, see Add specific version sources for production environments (CASE).
-
Add the IBM Operator Catalog to your cluster to make the latest version of Event Streams available in your cluster and for updates to be applied automatically. This option is suitable for nonproduction deployments where you always want to be on the latest version and are not concerned about an unexpected outage when Event Streams is updated automatically. For more information, see Add auto-updating sources for development or test environments (IBM Operator Catalog).
Add specific version sources for production environments (CASE)
Before you can install the required operator versions, make their catalog source available in your cluster as described in the following steps.
Note: This procedure must be done by using the CLI.
-
Install the IBM Catalog Management plug-in for IBM Cloud Paks plug-in. Run the following command to confirm that the plug-in is installed:
oc ibm-pak --help
-
Run the following command to download, validate, and extract the latest CASE version.
oc ibm-pak get ibm-eventstreams
Note: To install an earlier version of Event Streams, add the argument:
--version <case-version>
. See the support matrix to identify the CASE version that corresponds to your required Event Streams version. -
Generate mirror manifests by running the following command:
oc ibm-pak generate mirror-manifests ibm-eventstreams icr.io
The previous command generates the following files based on the target internal registry provided:
- catalog-sources.yaml
- catalog-sources-linux-
<arch>
.yaml (if there are architecture-specific catalog sources) - image-content-source-policy.yaml
- images-mapping.txt
-
Apply the catalog sources for the operator to the cluster by running the following command:
oc apply -f ~/.ibm-pak/data/mirror/ibm-eventstreams/<case-version>/catalog-sources.yaml
Where
<case-version>
is the version of the CASE that you want to install. Refer to the support matrix to identify the CASE version that corresponds to the Event Streams version that you want to install.
To make the Event Streams operator available in the OpenShift OperatorHub catalog, create the following YAML files and apply them as follows.
These steps add the catalog source for Event Streams, making the operator available to install.
Install the operator by using the OpenShift web console or the CLI.
Add auto-updating sources for development or test environments (IBM Operator Catalog)
Important: Use this method of installation only if you want Event Streams updates to be applied automatically when they become available.
Note: If you have other IBM products installed in your cluster, the IBM Operator Catalog might already be available. If it is configured for automatic updates as described in the following section, it already contains the required operators, and you can skip deploying the IBM Operator Catalog.
If you are installing the Event Streams operator as the first IBM operator in your cluster, create and apply the following YAML file to make the operators available in the OpenShift OperatorHub catalog.
To add the IBM Operator Catalog:
-
Create a file for the IBM Operator Catalog source with the following content, and save as
ibm_catalogsource.yaml
:apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ibm-operator-catalog namespace: openshift-marketplace spec: displayName: "IBM Operator Catalog" publisher: IBM sourceType: grpc image: icr.io/cpopen/ibm-operator-catalog updateStrategy: registryPoll: interval: 45m
Automatic updates of your IBM Operator Catalog can be disabled by removing the polling attribute,
spec.updateStrategy.registryPoll
. To disable automatic updates, remove the following parameters in the IBM Operator Catalog source YAML under thespec
field:updateStrategy: registryPoll: interval: 45m
Important: Other factors such as Subscription might enable the automatic updates of your deployments. For tight version control of your operators or to install a specific version use a CASE bundle instead of the IBM Operator catalog.
- Log in to your Red Hat OpenShift Container Platform as a cluster administrator by using the
oc
CLI (oc login
). -
Apply the source by using the following command:
oc apply -f ibm_catalogsource.yaml
Alternatively, you can add the catalog source through the OpenShift web console by using the Import YAML option:
- Select the plus icon on the upper right.
- Paste the IBM Operator Catalog source YAML in the YAML editor. You can also drag-and-drop the YAML files into the editor.
- Select Create.
These steps add the catalog source for Event Streams to the OperatorHub catalog, making the operator available to install.
Install the Event Streams operator
Ensure you have considered the Event Streams operator requirements, including resource requirements and the required cluster-scoped permissions.
Installing by using the web console
To install the operator by using the OpenShift Container Platform web console, complete the following steps:
- Log in to the OpenShift Container Platform web console using your login credentials.
- Expand the Operators dropdown and select OperatorHub to open the OperatorHub dashboard.
- Select the project you want to deploy the Event Streams instance in.
- In the All Items search box enter
Event Streams
to locate the operator title. - Click the Event Streams tile to open the install side panel.
- Click the Install button to open the Create Operator Subscription dashboard.
- Select the chosen installation mode that suits your requirements. If the installation mode is A specific namespace on the cluster, select the target namespace you created previously.
- Set Update approval to Automatic.
- Click Install to begin the installation.
The installation can take a few minutes to complete.
Important: Only install one Event Streams operator on a cluster.
Installing by using the command line
To install the operator by using the OpenShift Container Platform command line, complete the following steps:
-
Change to the namespace (project) where you want to install the operator. For command line installations, this sets the chosen installation mode for the operator:
- Change to the system namespace
openshift-operators
if you are installing the operator to be able to manage instances in all namespaces. - Change to the custom namespace if you are installing the operator for use in a specific namespace only.
oc project <target-namespace>
- Change to the system namespace
-
Check whether there is an existing
OperatorGroup
in your target namespace:oc get OperatorGroup
If there is an existing
OperatorGroup
, continue to the next step to create aSubscription
.If there is no
OperatorGroup
, create one as follows:a. Create a YAML file with the following content, replacing
<target-namespace>
with your namespace:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: ibm-eventautomation-operatorgroup namespace: <target-namespace> spec: targetNamespaces: - <target-namespace>
b. Save the file as
operator-group.yaml
.c. Run the following command:
oc apply -f operator-group.yaml
-
Create a
Subscription
for the Event Streams operator as follows:a. Create a YAML file similar to the following example:
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ibm-eventstreams namespace: <target-namespace> labels: backup.eventstreams.ibm.com/component: subscription spec: channel: <current-channel> installPlanApproval: Automatic name: ibm-eventstreams source: <catalog-source-name> sourceNamespace: openshift-marketplace
Where:
<target-namespace>
is the namespace where you want to install Event Streams (openshift-operators
if you are installing in all namespaces, or a custom name if you are installing in a specific namespace).<current-channel>
is the operator channel for the release you want to install (see the support matrix). For example v3.6.<catalog-source-name>
is the name of the catalog source that was created for this operator. This isibm-eventstreams
when installing a specific version by using a CASE bundle, oribm-operator-catalog
if the source is the IBM Operator Catalog.
b. Save the file as
subscription.yaml
.c. Run the following command:
oc apply -f subscription.yaml
Checking the operator status
You can view the status of the installed operator as follows.
-
To see the installed operator and check its status by using the web console, complete the following steps:
- Log in to the OpenShift Container Platform web console using your login credentials.
- Expand the Operators dropdown and select Installed Operators to open the Installed Operators page.
- Expand the Project dropdown and select the project the instance is installed in. Click the operator called IBM Event Streams managing the project.
- Scroll down to the ClusterServiceVersion details section of the page.
- Check the Status field. After the operator is successfully installed, this will change to
Succeeded
.
In addition to the status, information about key events that occur can be viewed under the Conditions section of the same page. After a successful installation, a condition with the following message is displayed:
install strategy completed with no errors
. -
To check the status of the installed operator by using the command line:
oc get csv
The command returns a list of installed operators. The installation is successful if the value in the
PHASE
column for your Event Streams operator isSucceeded
.
Note: If the operator is installed into a specific namespace, then it will only appear under the associated project. If the operator is installed for all namespaces, then it will appear under any selected project. If the operator is installed for all namespaces and you select all projects from the Project drop down, the operator will be shown multiple times in the resulting list, once for each project.
When the Event Streams operator is installed, the following additional operators will appear in the installed operator list:
- Operand Deployment Lifecycle Manager.
- IBM Common Service Operator.
Scaling the operator for high availability
High availability (HA) is the elimination of single points of failure in an environment. In addition to setting up your Kafka brokers for high availability, you can also set the number of the Event Streams operator replicas to enable more resilience.
By increasing the number of replicas to a value greater than 1, you can ensure that the Event Streams operator continues to function in a wider range of outage scenarios. To ensure uptime in failure situations, the management of your Event Streams is delegated to the other available operator pods.
To increase the number replicas, edit the replicas in the ClusterServiceVersion
object manually or by running the following command:
oc patch csv -n <NAMESPACE> ibm-eventstreams.v<CSV_VERSION> -p '[{"op":"replace","path":"/spec/install/spec/deployments/0/spec/replicas","value":3}]' --type json
Technology Preview feature: KRaft
Technology Preview features are available to evaluate potential upcoming features. Such features are intended for testing purposes only and not for production use. IBM does not support these features, but might help with any issues raised against them. IBM welcomes feedback on Technology Preview features to improve them. As the features are still under development, functions and interfaces can change, and it might not be possible to upgrade when updated versions become available.
IBM offers no guarantee that Technology Preview features will be part of upcoming releases and as such become fully supported.
Event Streams version 11.1.5 and later includes Apache Kafka Raft (KRaft) as a Technology Preview feature. KRaft replaces ZooKeeper for managing metadata, moving the overall handling of metadata into Kafka itself.
Limitations
The KRaft mode in Event Streams has the following limitations:
- Moving existing Kafka clusters deployed with ZooKeeper to use KRaft, or the other way around, is not supported.
- Upgrading your Apache Kafka or Event Streams operator version, or reverting either one to an earlier version is not supported. To do so, you delete the cluster, upgrade the operator, and deploy a new Kafka cluster.
- The Topic Operator is not supported. The
spec.entityOperator.topicOperator
property must be removed from the Kafka custom resource. - JBOD storage is not supported. You can use
type: jbod
for storage, but the JBOD array can contain only one disk. - Geo-replication is not supported.
Enabling KRaft
To enable KRaft, add the following annotations to the EventStreams
custom resource:
eventstreams.ibm.com/kraft: enabled
eventstreams.ibm.com/node-pools: enabled
For example:
- metadata:
name: light-insecure
namespace: openshift-operators
annotations:
eventstreams.ibm.com/node-pools: enabled
eventstreams.ibm.com/kraft: enabled
Install an Event Streams instance
Instances of Event Streams can be created after the Event Streams operator is installed. If the operator was installed into a specific namespace, then it can only be used to manage instances of Event Streams in that namespace. If the operator was installed for all namespaces, then it can be used to manage instances of Event Streams in any namespace, including those created after the operator was deployed.
When installing an instance of Event Streams, ensure you are using a namespace that an operator is managing.
Installing an instance by using the web console
To install an Event Streams instance through the OpenShift Container Platform web console, do the following:
- Log in to the OpenShift Container Platform web console using your login credentials.
- Expand the Operators dropdown and select Installed Operators to open the Installed Operators page.
-
Expand the Project dropdown and select the project the instance is installed in. Click the operator called IBM Event Streams managing the project.
Note: If the operator is not shown, it is either not installed or not available for the selected namespace.
- In the Operator Details dashboard, click the Event Streams tab.
- Click the Create EventStreams button to open the Create EventStreams panel. You can use this panel to define an
EventStreams
custom resource.
From here you can install by using the form view. For more advanced configurations or to install one of the samples, see installing by using the YAML view.
Installing by using the form view
To configure an EventStreams
custom resource, do the following:
- Enter a name for the instance in the Name field.
- Click the license accept toggle to set it to True.
- Ensure that the correct values for Product License and Product Use are selected from the drop-down lists. These values are used for metering purposes and could result in inaccurate charging and auditing if set incorrectly. For more information about the available options, see the licensing reference.
- Optional: You can configure other components such as Kafka, ZooKeeper, and Security to suit your requirements.
- Scroll down and click the Create button at the bottom of the page to deploy the Event Streams instance.
- Wait for the installation to complete.
- You can now verify your installation and consider other post-installation tasks.
Installing by using the YAML view
Alternatively, you can configure the EventStreams
custom resource by editing YAML documents. To do this, click the Edit YAML tab.
A number of sample configurations are provided on which you can base your deployment. These range from smaller deployments for non-production development or general experimentation to large scale clusters ready to handle a production workload. Alternatively, a pre-configured YAML file containing the custom resource sample can be dragged and dropped onto this screen to apply the configuration.
To view the samples, do the following:
- Select the Samples tab to show the available sample configurations.
- Click the Try it link under any of the samples to open the configuration in the Create EventStreams panel.
More information about these samples is available in the planning section. You can base your deployment on the sample that most closely reflects your requirements and apply customizations on top as required.
When modifying the sample configuration, the updated document can be exported from the Create EventStreams panel by clicking the Download button and re-imported by dragging the resulting file back into the window.
Important: Ensure that the spec.license.accept
field in the custom resource YAML is set to true
, and that the correct values are selected for the spec.license.license
and spec.license.use
fields before deploying the Event Streams instance. These values are used for metering purposes and could result in inaccurate charging and auditing if set incorrectly. For more information about the available options, see the licensing reference.
Note: If experimenting with Event Streams for the first time, the Lightweight without security sample is the smallest and simplest example that can be used to create an experimental deployment. For the smallest production setup, use the Minimal production sample configuration.
To deploy an Event Streams instance, use the following steps:
- Complete any changes to the sample configuration in the Create EventStreams panel.
- Click Create to begin the installation process.
- Wait for the installation to complete.
- You can now verify your installation and consider other post-installation tasks.
Installing an instance by using the CLI
To install an instance of Event Streams from the command-line, you must first prepare an EventStreams
custom resource configuration in a YAML file.
A number of sample configuration files are available in GitHub, where you can select the GitHub tag for your Event Streams version to access the correct samples, and then go to /cr-examples/eventstreams/openshift
to access the OpenShift samples.
The sample configurations range from smaller deployments for non-production development or general experimentation to large scale clusters ready to handle a production workload.
More information about these samples is available in the planning section. You can base your deployment on the sample that most closely reflects your requirements and apply customizations on top as required.
Important: Ensure that the spec.license.accept
field in the custom resource YAML is set to true
, and that the correct values are selected for the spec.license.license
and spec.license.use
fields before deploying the Event Streams instance. These values are used for metering purposes and could result in inaccurate charging and auditing if set incorrectly. For more information about the available options, see the licensing reference.
Note: If experimenting with Event Streams for the first time, the Lightweight without security sample is the smallest and simplest example that can be used to create an experimental deployment. For the smallest production setup, use the Minimal production sample configuration.
To deploy an Event Streams instance, run the following commands:
-
Set the project where your
EventStreams
custom resource will be deployed in:oc project <project-name>
-
Apply the configured
EventStreams
custom resource:oc apply -f <custom-resource-file-path>
For example:
oc apply -f development.yaml
- Wait for the installation to complete.
- You can now verify your installation and consider other post-installation tasks.