Ensure your environment meets the following prerequisites before installing Event Streams.
Container environment
Event Streams 2019.2.1 is supported on the following platforms and systems:
| Container platform | Systems |
|---|---|
| Red Hat OpenShift Container Platform 3.11 with IBM cloud foundational services 3.2.0.1907* (or later fix pack) and 3.2.1 * | - Linux® 64-bit (x86_64) systems - Microsoft Azure (IaaS) - Amazon Web Services (IaaS) |
| Red Hat OpenShift Container Platform 3.10 with IBM Cloud Private 3.1.2* | - Linux® 64-bit (x86_64) systems - Microsoft Azure (IaaS) - Amazon Web Services (IaaS) |
| IBM Cloud Private 3.1.2, 3.2.0.1907 (or later fix pack), and 3.2.1 | - Linux® 64-bit (x86_64) systems - Linux on IBM® z13 or later systems - Microsoft Azure (IaaS) - Amazon Web Services (IaaS) |
*Provided by IBM Cloud Private
Event Streams 2019.2.1 has Helm chart version 1.3.0 and includes Kafka version 2.2.0. For an overview of supported component and platform versions, see the support matrix.
Ensure you have the following set up for your environment:
- If you are installing Event Streams on the OpenShift Container Platform, ensure you have the right version of OpenShift installed and integrated with the right version of IBM Cloud Private. See previous table for supported versions. For example, install OpenShift 3.11, and integrate it with IBM Cloud Private 3.2.0.
-
Install and configure IBM Cloud Private.
Important: In high throughput environments, ensure you configure your IBM Cloud Private cluster to include an external load balancer and an internal network. These configuration options help take full advantage of Event Streams scaling and Kafka settings, and avoid potential performance bottlenecks. For more information, see the performance planning topic.Note: Event Streams includes entitlement to IBM Cloud Private Foundation which you can download from IBM Passport Advantage.
- If you are installing Event Streams on an IBM Cloud Private cluster deployed on Amazon Web Services (AWS), ensure your proxy address uses lowercase characters.
- If you are installing Event Streams on an IBM Cloud Private cluster deployed on Microsoft Azure, ensure you first register a Service Principal (an application in the Azure Active Directory). For information about creating a Service Principal, see the terraform documentation.
- Install the Kubernetes command line tool, and configure access to your cluster.
- If you are installing Event Streams on the OpenShift Container Platform, ensure you also install the OpenShift Container Platform CLI.
- Install the IBM Cloud Private Command Line Interface (CLI).
- Install the Helm CLI required for your version of IBM Cloud Private, and add the IBM Cloud Private internal Helm repository called
local-chartsto the Helm CLI as an external repository. -
For message indexing capabilities (enabled by default), ensure you set the
vm.max_map_countproperty to at least262144on all IBM Cloud Private nodes in your cluster (not only the master node). Run the following commands on each node:
sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | tee -a /etc/sysctl.confImportant: This property might have already been updated by other workloads to be higher than the minimum required.
Hardware requirements
The Helm chart for Event Streams specifies default values for the CPU and memory usage of the Apache Kafka brokers and Apache ZooKeeper servers.
See the following table for memory requirements of each Helm chart component. Ensure you have sufficient physical memory to service these requirements.
Kubernetes manages the allocation of containers within your cluster. This allows resources to be available for other Event Streams components which might be required to reside on the same node.
Ensure you have one IBM Cloud Private worker node per Kafka broker, and a minimum of 3 worker nodes available for use by Event Streams. Ensure each worker node runs on a separate physical server. See the guidance about Kafka high availability for more information.
Helm resource requirements
The Event Streams helm chart has the following resource requirements based on resource request and limit settings. Requests and limits are Kubernetes concepts for controlling resource types such as CPU and memory.
- Requests set the minimum requirements a container requires to be scheduled. If your system does not have the required request value, then your services will not start up.
- Limits set the value beyond which a container cannot consume the resource. It is the upper limit within your system for the service.
For more information about resource requests and limits, see the Kubernetes documentation.
The following table lists the aggregate resource requirements of the Event Streams helm chart. The table includes totals for both request and limit values of all pods and their containers. Each container in a pod has its own request and limit values, but as pods run as a group, these values need to be added together to understand the total requirements for a pod. For details about the requirements for each container within individual pods, see the tables in the following sections.
These are the minimum requirements for an Event Streams installation, and will be used unless you change them when configuring your installation. They are based on the default resource request and limit settings of the chart. Installing with these settings is suitable for a starter deployment intended for testing purposes and trying out Event Streams.
For a production setup, ensure you set higher values, and also consider important configuration options for IBM Cloud Private such as setting up a load balancer and an internal network. For more information about planning for a production setup, including requirements for a baseline production environment, see the performance planning topic.
| Pod group | Configurable replicas | Total CPU request per pod group (cores) | Total CPU limit per pod group (cores) | Total memory request per pod group (Gi) | Total memory limit per pod group (Gi) |
|---|---|---|---|---|---|
| Kafka | 3* | 9.6* (3.2 per broker) | 11.4* (3.8 per broker) | 14.4* (4.8 per broker) | 14.4* (4.8 per broker) |
| Event Streams core | 5.1 without persistence | 12.4 without persistence | 7.7 without persistence | 10.6 without persistence | |
| 6.1 with persistence enabled | 13.4 with persistence enabled | 8.2 with persistence enabled | 11.1 with persistence enabled | ||
| Message indexing | 1.5 | 2.5 | 4.4 | 8.4 | |
| Geo-replication | 0* | 0.9 per replica | 1.6 per replica | 2.5 per replica | 2.5 per replica |
| TOTAL FOR EVENT STREAMS | 17.1 without persistence | 27.9 without persistence | 29 without persistence | 35.9 without persistence | |
| 18.1 with persistence enabled | 28.9 with persistence enabled | 29.5 with persistence enabled | 36.4 with persistence enabled |
Important: The settings marked with an asterisk (*) are configurable. The values in the table are the default minimum values.
Before installing Event Streams (not Community Edition), consider the number of Kafka replicas and geo-replicator nodes you plan to use. Each Kafka replica and geo-replicator node is a separate chargeable unit.
Note: Geo-replication is only available in the paid-for version of Event Streams (not available in the Community Edition).
Kafka group
The following pods and their containers are part of this group.
Important: The settings marked with an asterisk (*) are configurable. The values in the table are the default minimum values.
Kafka pod
Number of replicas: 3*
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Kafka | 1* | 1* | 2* | 2* |
| Metrics reporter | 0.4 | 0.6 | 1.5 | 1.5 |
| Metrics proxy | 0.5 | 0.5 | 1 | 1 |
| Healthcheck | 0.2 | 0.2 | 0.1 | 0.1 |
| TLS proxy | 0.1 | 0.5 | 0.1 | 0.1 |
Network proxy pod
Number of replicas: 3 (matches the number of Kafka replicas)
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Proxy | 1 | 1 | 0.1 | 0.1 |
Event Streams core group
The following pods and their containers are part of this group.
Important: The settings marked with an asterisk (*) are configurable. The values in the table are the default minimum values.
ZooKeeper pod
Number of replicas: 3
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| ZooKeeper | 0.1* | 0.1* | 0.75 | 1 |
| TLS proxy | 0.1 | 0.1 | 0.1 | 0.1 |
Administration UI pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| UI | 1 | 1 | 1 | 1 |
| Redis | 0.1 | 0.1 | 0.1 | 0.1 |
Administration server pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Rest | 0.5 | 4 | 1 | 2 |
| Codegen | 0.2 | 0.5 | 0.3 | 0.5 |
| TLS proxy | 0.1 | 0.1 | 0.1 | 0.1 |
REST producer server pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Rest-producer | 0.5 | 4 | 1 | 2 |
REST proxy pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Rest-proxy | 0.5 | 0.5 | 0.25 | 0.25 |
Collector pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Collector | 0.1 | 0.1 | 0.05 | 0.05 |
| TLS proxy | 0.1 | 0.1 | 0.1 | 0.1 |
Access controller pod
Number of replicas: 2
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Access controller | 0.1 | 0.1 | 0.25 | 0.25 |
| Redis | 0.1 | 0.1 | 0.1 | 0.1 |
Schema Registry pod
Number of replicas:
- 1 without persistence
- 2 with persistence enabled
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Schema Registry | 0.5 | 0.5 | 0.25 | 0.25 |
| Avro service | 0.5 | 0.5 | 0.25 | 0.25 |
Message indexing group
The following pods and their containers are part of this group.
Index manager pod
Number of replicas: 1
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Index manager | 0.2 | 0.2 | 0.1 | 0.1 |
| TLS proxy | 0.1 | 0.1 | 0.1 | 0.1 |
Elasticsearch pod
Number of replicas: 2
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Elastic | 0.5* | 1* | 2* | 4* |
| TLS proxy | 0.1 | 0.1 | 0.1 | 0.1 |
Geo-replicator group
This group only contains the geo-replicator pod.
Number of replicas: 0*
Note: This means there is no geo-replication enabled by default. The values in the table are the default minimum values for 1 replica.
| Container | CPU request per container (cores) | CPU limit per container (cores) | Memory request per container (Gi) | Memory limit per container (Gi) |
|---|---|---|---|---|
| Replicator | 0.5 | 1 | 1 | 1 |
| Metrics reporter | 0.4 | 0.6 | 1.5 | 1.5 |
PodSecurityPolicy requirements
To install the Event Streams chart, you must have the ibm-restricted-psp PodSecurityPolicy selected for the target namespace.
You can define the PodSecurityPolicy when creating the namespace for your installation.
Event Streams applies network policies to control the traffic within the namespace where it is deployed, limiting the traffic to that required by Event Streams. For more information about the network policies and the traffic they permit, see network policies.
For more information about PodSecurityPolicy definitions, see the IBM Cloud Private documentation.
Note: The PodSecurityPolicy requirements do not apply to the Red Hat OpenShift Container Platform.
Red Hat OpenShift SecurityContextConstraints Requirements
If you are installing on the OpenShift Container Platform, the Event Streams chart requires a custom SecurityContextConstraints to be bound to the target namespace prior to installation. The custom SecurityContextConstraints controls the permissions and capabilities required to deploy this chart.
You can enable this custom SecurityContextConstraints resource using the supplied pre-installation setup script.
Network requirements
Event Streams is supported for use with IPv4 networks only.
File systems for storage
If you want to set up persistent storage, you must have physical volumes available, backed by one of the following file systems:
- NFS version 4
- GlusterFS version 3.10.1
- IBM Spectrum Scale version 5.0.3.0
Event Streams user interface
The Event Streams user interface (UI) is supported on the following web browsers:
- Google Chrome version 65 or later
- Mozilla Firefox version 59 or later
- Safari version 11.1 or later
Event Streams CLI
The Event Streams command line interface (CLI) is supported on the following systems:
- Windows 10 or later
- Linux® Ubuntu 16.04 or later
- macOS 10.13 (High Sierra) or later
Clients
The Apache Kafka Java client included with Event Streams is supported for use with the following Java versions:
- IBM Java 8
- Oracle Java 8
You can also use other Kafka version 2.0 or later clients when connecting to Event Streams. If you encounter client-side issues, IBM can assist you to resolve those issues (see our support policy).
Event Streams is designed for use with clients based on the librdkafka implementation of the Apache Kafka protocol.