Prerequisites

Ensure your environment meets the following prerequisites before installing IBM Operator for Apache Flink and Event Processing.

Container environment

Event Processing 1.1.x is supported on the Red Hat OpenShift Container Platform and the other Kubernetes platforms that support the Red Hat Universal Base Images (UBI) containers.

If you are using Red Hat OpenShift Container Platform, ensure you have the following set up for your environment:

• A supported version of the OpenShift Container Platform installed. For supported versions, see the support matrix.
• The OpenShift Container Platform CLI (oc) installed.

If you are using other Kubernetes platforms, ensure you have the following set up for your environment:

• A supported version of a Kubernetes platform installed. For supported versions, see the support matrix.
• The Kubernetes command-line tool (kubectl) installed.

Hardware requirements

Ensure that your hardware can accommodate the resource requirements for your planned deployment.

Resource requirements

Event Processing resource requirements depend on several factors. The following sections provide guidance about minimum requirements for a starter deployment, and options for initial production configurations.

Minimum resource requirements are based on the total of requests set for the deployment. You will require more resources to accommodate the limit settings (see more about “requests” and “limits” later in this section). Always ensure that you have sufficient resources in your environment to deploy the IBM Operator for Apache Flink and Event Processing operator together with a Quick start or a Production instance of Flink and Event Processing.

Flink resource requirements:

Event Processing resource requirements:

Note: Event Processing provides sample configurations to help you get started with deployments. The resource requirements for these specific samples are detailed in the planning section. If you do not have an Event Processing installation on your system yet, always ensure that you include the resource requirements for the operator together with the intended Event Processing instance requirements (quick start or production).

Note: IBM Operator for Apache Flink When deploying a FlinkDeployment custom resource, the initial number of Flink Job Manager (JM) and Task Manager (TM) pods is equal to the number of replicas indicated for each in the custom resource. All the provided samples configure 1 replica for both JM and TM, except the “Production” sample which has 2 replicas for JM. Additional Task Manager pods are created if the number of deployed Flink jobs exceeds the TM “slot capacity”, which is determined by the parameter spec.flinkConfiguration.taskmanager.numberOfTaskSlots in the FlinkDeployment. When running Event Processing flows, each flow generates 2 Flink jobs; each job is allocated its own Task Manager slot.

Event Processing is a Kubernetes operator-based release and uses custom resources to define your Event Processing configurations. The Event Processing operator uses the declared required state of your Event Processing in the custom resources to deploy and manage the entire lifecycle of your Event Processing instances. Custom resources are presented as YAML configuration documents that define instances of the EventProcessing custom resource type.

The provided samples define typical configuration settings for your Event Processing instance, including security settings, and default values for resources such as CPU and memory defined as “request” and “limit” settings. Requests and limits are Kubernetes concepts for controlling resource types such as CPU and memory.

• Requests set the minimum requirements that a container requires to be scheduled. If your system does not have the required request value, then the services do not start.
• Limits set the value beyond which a container cannot consume the resource. It is the upper limit within your system for the service. Containers that exceed a CPU resource limit are throttled, and containers that exceed a memory resource limit are terminated by the system.

Ensure that you have sufficient CPU capacity and physical memory in your environment to service these requirements. Your Event Processing instance can be updated dynamically through the configuration options that are provided in the custom resource.

Operator requirements

The Event Processing operator and the IBM Operator for Apache Flink have the following requirements.

Requirements for the Event Processing operator

The Event Processing operator has the following minimum resource requirements. Ensure that you always include sufficient CPU capacity and physical memory in your environment to service the operator requirements.

Cluster-scoped permissions required

The Event Processing operator requires the following cluster-scoped permissions:

• Permission to manage admission webhooks: The Event Processing operator uses admission webhooks to provide immediate validation and feedback about the creation and modification of Event Processing instances. The permission to manage webhooks is required for the operator to register these actions.
• Permission to manage ConsoleYAMLSamples: ConsoleYAMLSamples are used to provide samples for Event Processing resources in the OpenShift Container Platform web console. The permission to manage ConsoleYAMLSamples is required for the operator to register the setting up of samples.
• Permission to list specific CustomResourceDefinitions: This allows Event Processing to identify whether other optional dependencies have been installed into the cluster.

In addition to the previous permissions, the Event Processing operator requires the following cluster-scoped permissions on Red Hat OpenShift Container Platform:

• Permission to list ClusterRoles and ClusterRoleBindings: The Event Processing operator uses ClusterRoles created by the Operator Lifecycle Manager (OLM) as parents for supporting resources that the Event Processing operator creates. This is needed so that the supporting resources are correctly cleaned up when Event Processing is uninstalled. The permission to list ClusterRoles is required to allow the operator to identify the appropriate cluster role to use for this purpose.

Operator requirements for the IBM Operator for Apache Flink

The IBM Operator for Apache Flink has the following minimum resource requirements.

You can only install one version of the IBM Operator for Apache Flink on a cluster. Installing multiple versions on a single cluster is not supported.

You cannot install the IBM Operator for Apache Flink on a cluster that already has the open-source Apache Flink operator installed. If the Apache Flink operator is already installed, ensure you uninstall it first, including the removal of related Custom Resource Definitions (CRDs), and then install the IBM Operator for Apache Flink.

Red Hat OpenShift Security context constraints

Event Processing requires a security context constraint (SCC) to be bound to the target namespace prior to installation.

By default, Event Processing complies with restricted or restricted-v2 SCC depending on your OpenShift Container Platform version.

Network requirements

Event Processing is supported for use with IPv4 networks only.

Ingress controllers

To expose Event Processing services externally outside your cluster, the Event Processing operator will create:

• OpenShift routes when installing on Red Hat OpenShift Container Platform.

• Kubernetes ingress resources when installing on other Kubernetes platforms.

To use ingress, ensure you install and run an ingress controller on your Kubernetes platform. The SSL passthrough must be enabled in the ingress controller for your Event Processing services to work. Refer to your ingress controller documentation for more information.

Data storage requirements

If you want to set up persistent storage, see the planning section.

Storage requirements for Event Processing

Ensure that the cluster administrator creates one or more storage class that supports ReadWriteOnce and allows read and write access to non-root users.

For example, you can use one of the following systems:

• Red Hat OpenShift Data Foundation (previously OpenShift Container Storage) version 4.2 or later (block storage only)
• IBM Cloud Block storage
• IBM Storage Suite for IBM Cloud Paks: block storage from IBM Spectrum Virtualize, FlashSystem, or DS8K
• Portworx Storage version 2.5.5 or later
• Rook Ceph

If installing on RedHat OpenShift Kubernetes Service on IBM Cloud (ROKS), you can use either:

• IBM Cloud Block storage, ibmc-block-<tier>, where <tier> can be gold, silver or bronze
• IBM Cloud File storage with support for supplemental group IDs, ibmc-file-<tier>-gid (only on single-zone OpenShift clusters)

Storage requirements for Flink

The Flink instances deployed by IBM Operator for Apache Flink store the following data if persistent storage is configured:

• Checkpoints and savepoints. For more information about checkpoints, see the Flink documentation about checkpoint storage and checkpointing prerequisites. For more information about savepoints, see the Flink documentation.
• When configured to persist states in RocksDB, the data of processed events is stored in a binary, compressed, and unencrypted format.

Apache Flink requires the use of a persistent volume with the following capabilities:

• volumeMode: Filesystem. See Volume Mode.
• accessMode: ReadWriteMany (RWX). See access modes.

For example, you can use Rook Ceph for your storage.

If installing on RedHat OpenShift Kubernetes Service on IBM Cloud (ROKS), you can use either:

• IBM Cloud Block storage, ibmc-block-<tier>, where <tier> can be gold, silver or bronze
• IBM Cloud File storage with support for supplemental group IDs, ibmc-file-<tier>-gid (only on single-zone OpenShift clusters)

Important:

• For security reasons, the use of hostPath volumes is not supported.
• On the OpenShift Container Platform, the dynamically provisioned volumes are created with the reclaim policy set to Delete by default. This means that the volume lasts only while the claim still exists in the system. If you delete the claim, the volume is also deleted, and all data on the volume is lost. If this is not appropriate, you can use the Retain policy. For more information about the reclaim policy, see the Kubernetes documentation.
• Some storage classes support dynamic provisioning. If the StorageClass that you use supports dynamic provisioning, then a PersistentVolume can be dynamically provisioned when the PersistentVolumeClaim is created. Otherwise, the PersistentVolume must be created before you define the PersistentVolumeClaim.

Event Processing UI

The Event Processing user interface (UI) is supported on the following web browsers:

• Google Chrome version 113 or later
• Mozilla Firefox version 113 or later
• Safari version 16.5 or later

Certificate management

By default, all certificates that are required by Event Processing are managed by a certificate manager. The certificate manager simplifies the process of creating, renewing, and using those certificates.

• On Red Hat OpenShift Container Platform, install IBM Cert Manager.
• On other Kubernetes platforms, use a Cert Manager that supports GroupVersionKind (GVK) such as Issuer.cert-manager.io/v1 and Certificate.cert-manager.io/v1, or create certificates manually and provide them to Event Processing by using Kubernetes secrets.

IBM Cert Manager on Red Hat OpenShift Container Platform

If you already have a Cert Manager installed in your cluster, you can skip this installation.

To check whether a Cert Manager is installed in your cluster by using the OpenShift web console, complete the following steps:

1. Log in to the OpenShift Container Platform web console using your login credentials.
2. Expand the Operators dropdown and select Installed Operators to open the Installed Operators page.
3. In the list of installed operators, check whether IBM Cert Manager is available and status is showing as Succeeded.

Alternatively, to check whether Cert Manager is installed in your cluster by using the CLI, run the following command and check whether ibm-cert-manager-operator is available:

oc get subs --all-namespaces

Important: You can only have one Cert Manager operator installed in your cluster. Choose the appropriate version depending on what other software is running in your environment. If you have an existing IBM Cloud Pak for Integration in your environment, check whether you have the foundational services operator running already and note the version.

Follow the instructions to install the Cert Manager based on the foundational services version.

With foundational services version 3.x

If you are installing IBM Operator for Apache Flink and Event Processing on a cluster where foundational services version 3.x is already installed, follow the instructions to install the Cert Manager operator.

Note: The following instructions apply in both online and offline clusters.

1. Ensure you have installed the foundational services as described in the foundational services documentation.

2. Create an OperandRequest custom resource with the following YAML in the Operand Deployment Lifecycle Manager as described in the foundational services documentation.

   apiVersion: operator.ibm.com/v1alpha1
   kind: OperandRequest
   metadata:
      name: common-service
      namespace: <namespace>
   spec:
   requests:
      - operands:
        - name: ibm-cert-manager-operator
        registry: common-service
        registryNamespace: ibm-common-services
   

   Where <namespace> is the namespace from where you are creating the OperandRequest and planning to install the Event Processing operator.

Verify that the cert-manager operator shows as Succeeded in the namespaces where you want to use Event Processing.

With foundational services version 4.0.0, or without foundational services

Follow the instructions to install the Cert Manager 4.0.0 as part of foundational services version 4.0.0, or without foundational services in an online or offline environment.

For online environments

Follow the foundational services instructions to install the Cert Manager version 4.0.0 in online environments that are running with or without foundational services.

Verify that the cert-manager operator shows as Succeeded in the namespaces where you want to use Event Processing.

For offline environments

If you are installing Event Processing in an offline environment, follow the instructions in the foundational services documentation to mirror the necessary images, and install the Cert Manager from its CASE bundle.

Verify that the cert-manager operator shows as Succeeded in the namespaces where you want to use Event Processing.

Schema registry requirements

You can define the event structure with an Avro schema that uses a schema registry.

Before you configure the event source node with an Avro schema that uses a schema registry, the following requirements must be met:

• Only the schema registry from Event Streams or a registry that supports the Confluent REST API is supported.
• To securely connect the schema registry and Event Processing, an SSL connection must be enabled. For more information, see configuring.

• Ensure that you retrieve the URL to the REST endpoint of the schema registry. For example, to retrieve the URL from Event Streams, complete the following steps:

  1. Log in to your Event Streams UI as an administrator from a supported web browser. For more information, see how to determine the login URL for your Event Streams UI.
  2. In the Event Streams home page, click the Connect to this cluster tile to open the Cluster connection tile.
  3. Copy the schema registry URL in the Schema registry endpoint. Use this URL when you configure the event source node.