IBM Event Streams provides a high-throughput producer application you can use as a workload generator to test message loads and help validate the performance capabilities of your cluster.
You can use one of the predefined load sizes, or you can specify your own settings to test throughput. Then use the test results to ensure your cluster setup is appropriate for your requirements, or make changes as needed, for example, by changing your scaling settings.
You can download the latest pre-built producer application.
If you cloned the Git repository, build the producer as follows:
- Ensure you have Java version 8. Also, ensure you have Apache Maven installed on your system.
- Ensure you have cloned the Git project.
- Open a terminal and change to the root directory.
- Run the following command:
You can also specify your root directory using the
-foption as follows
mvn install -f <path_to>/pom.xml
es-producer.jarfile is created in the
The producer application requires configuration settings that you can set in the provided
producer.config template configuration file.
producer.config file is located in the root directory. If you downloaded the pre-built producer, you have to run the
es-producer.jar with the
-g option to generate the configuration file. If you build the producer application yourself, the configuration file is created and placed in the root for you when building.
Before running the producer to test loads, you must specify the following details in the configuration file.
||The URL used for bootstrapping knowledge about the rest of the cluster. You can find this address in the Event Streams UI as described later.|
||The location of the JKS keystore used to securley communicate with your IBM Event Streams instance. You can downloaded the JKS keystore file from the Event Streams UI as described later.|
||An API key is needed to authorize production to your topic, and must be added to the ‘password’ string. To generate API keys, go to the Event Streams UI as described later.|
Obtaining configuration details
Obtain the required configuration details as follows:
- Log in to your IBM Event Streams UI.
- Click Connect to this cluster on the right.
- Go to the Connect a client tab.
- Locate the details:
- For the
bootstrap.servers, copy the address from the Bootstrap server section.
- To downloaded the JKS keystore file, go to the Certificates section, and download the server certificate from the Java truststore section. Set the
ssl.truststore.locationto the full path and name of the downloaded file.
- To generate API keys, go to the API key section and follow the instructions.
- For the
You can secure access to your topics as described in managing access.
Create a load on your IBM Event Streams Kafka cluster by running the
es-producer.jar command. You can specify the load size based on the provided predefined values, or you can provide specific values for throughput and total messages to determine a custom load.
Using predefined loads
To use a predefined load size from the producer application, use the
es-producer.jar with the
java -jar target/es-producer.jar -t <topic-name> -s <small/medium/large>
For example, to create a
large message load based on the predefined
large load size, run the command as follows:
java -jar target/es-producer.jar -t myTopic -s large
This example creates a
large message load, where the producer attempts to send a total of 6,000,000 messages at a rate of 100,000 messages per second to the topic called
The following table lists the predefined load sizes the producer application provides.
|Size||Messages per second||Total messages|
Using user-defined loads
You can generate a custom message load using your own settings.
For example, to test the load to the topic called
myTopic with custom settings that create a total load of 60,000 messages with a size of 1024 bytes each, at a maximum throughput rate of 1000 messages per second, use the
es-producer.jar command as follows:
java -jar target/es-producer.jar -t myTopic -T 1000 -n 60000 -r 1024
The following table lists all the parameter options for the
||The name of the topic to send the produced message load to.||
||The total number of messages to be sent as part of the load. Note: The
||File to read the message payloads from. This works only for UTF-8 encoded text files. Payloads are read from this file and a payload is randomly selected when sending messages.|
||Provides delimiter to be used when
||Throttle maximum message throughput to approximately THROUGHPUT messages per second. -1 sets it to as fast as possible. Note: The
||Path to the producer configuration file.||
||Set whether to print out metrics at the end of the test.||
||The number of producer threads to run.||
||Pre-defined combinations of message throughput and volume. If used, this option overrides any settings specified by the
||The size of each message to be sent in bytes.||
||Lists the available parameters.|
||Generates the configuration file required to run the tool (
Note: You can override the parameter values by using the environment variables listed in the following table. This is useful, for example, when using containerization, and you are unable to specify parameters on the command line.
Note: If you set the size using
-s when running
es-producer.jar, you can only override it if both the
ES_THROUGHPUT environment variables are set, or if
ES_SIZE is set.