You can modify the capacity of your Event Streams system in a number of ways. See the following sections for details about the different methods, and their impact on your installation.
The pre-requisite guidance gives various examples of different production configurations on which you can base your deployment. To verify it meets your requirements, you should test the system with a workload that is representative of the expected throughput. For this purpose, Event Streams provides a workload generator application to test different message loads.
If this testing shows that your system does not have the capacity needed for the workload, whether this results in excessive lag or delays, or more extreme errors such as OutOfMemory
errors, then you can incrementally make the increases detailed in the following sections, re-testing after each change to identify a configuration that meets your specific requirements.
A performance report based on example case studies is also available to provide guidance for setting these values.
Note: Although the testing for the report was based on Apache Kafka version 2.3.0, the performance numbers are broadly applicable to current versions of Kafka as well.
Modifying the settings
These settings are defined in the EventStreams
custom resource under the spec.strimziOverrides
property. For more information on modifying these settings see modifying installation.
Increase the number of Kafka brokers in the cluster
The number of Kafka brokers is defined in the EventStreams
custom resource in the spec.strimziOverrides.kafka.replicas
property. For example to configure Event Streams to use 6 Kafka brokers:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
strimziOverrides:
# ...
kafka:
# ...
replicas: 6
Increase the CPU request or limit settings for the Kafka brokers
The CPU settings for the Kafka brokers are defined in the EventStreams
custom resource in the requests
and limits
properties under spec.strimziOverrides.kafka.resources
. For example to configure Event Streams Kafka brokers to have a CPU request set to 2 CPUs and limit set to 4 CPUs:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
strimziOverrides:
# ...
kafka:
# ...
resources:
requests:
cpu: 2000m
limits:
cpu: 4000m
A description of the syntax for these values can be found in the Kubernetes documentation.
Increase the memory request or limit settings for the Kafka brokers and ZooKeeper nodes
The memory settings for the Kafka brokers are defined in the EventStreams
custom resource in the requests
and limits
properties under spec.strimziOverrides.kafka.resources
.
The memory settings for the ZooKeeper nodes are defined in the EventStreams
custom resource in the requests
and limits
properties under spec.strimziOverrides.zookeeper.resources
.
For example to configure Event Streams Kafka brokers and ZooKeeper nodes to have a memory request set to 4GB
and limit set to 8GB
:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
strimziOverrides:
# ...
kafka:
# ...
resources:
requests:
memory: 4096Mi
limits:
memory: 8096Mi
zookeeper:
# ...
resources:
requests:
memory: 4096Mi
limits:
memory: 8096Mi
The syntax for these values can be found in the Kubernetes documentation.
Modifying the resources available to supporting components
The resource settings for each supporting component are defined in the EventStreams
custom resource in their corresponding component key the requests
and limits
properties under spec.<component>.resources
.
For example, to configure the Apicurio Registry to have a memory request set to 4GB
and limit set to 8GB
:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
apicurioRegistry:
# ...
resources:
requests:
memory: 4096Mi
limits:
memory: 8096Mi
The syntax for these values can be found in the Kubernetes documentation.
Modifying the JVM settings for Kafka brokers
If you have specific requirements, you can modify the JVM settings for the Kafka brokers.
Note: Take care when modifying these settings as changes can have an impact on the functioning of the product.
Note: Only a selected subset of the available JVM options can be configured.
JVM settings for the Kafka brokers are defined in the EventStreams
custom resource in the spec.strimziOverrides.kafka.jvmOptions
property. For example:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
strimziOverrides:
# ...
kafka:
# ...
jvmOptions:
-Xms: 4096m
-Xmx: 4096m
Increase the disk space available to each Kafka broker
The Kafka brokers need sufficient storage to meet the retention requirements for all of the topics in the cluster. Disk space requirements grow with longer retention periods for messages, increased message sizes and additional topic partitions.
The amount of storage made available to Kafka brokers is defined at the time of installation in the EventStreams
custom resource in the spec.strimziOverrides.kafka.storage.size
property. For example:
apiVersion: eventstreams.ibm.com/v1beta2
kind: EventStreams
# ...
spec:
# ...
strimziOverrides:
# ...
kafka:
# ...
storage:
# ...
size: 100Gi