Creating and testing message loads

Event Streams provides a high-throughput producer application you can use as a workload generator to test message loads and help validate the performance capabilities of your cluster.

You can use one of the predefined load sizes, or you can specify your own settings to test throughput. Then use the test results to ensure your cluster setup is appropriate for your requirements, or make changes as needed, for example, by changing your scaling settings.

Downloading

You can download the latest pre-built producer application.

Alternatively, you can clone the project from GitHub. However, if you clone from GitHub, you have to build the producer.

Building

If you cloned the Git repository, build the producer as follows:

1. Ensure you have Java version 8. Also, ensure you have Apache Maven installed on your system.
2. Ensure you have cloned the Git project.
3. Open a terminal and change to the root directory.
4. Run the following command: mvn install.
   You can also specify your root directory using the -f option as follows mvn install -f <path_to>/pom.xml
5. The es-producer.jar file is created in the /target directory.

Configuring

The producer application requires configuration settings that you can set in the provided producer.config template configuration file.

Note: The producer.config file is located in the root directory. If you downloaded the pre-built producer, you have to run the es-producer.jar with the -g option to generate the configuration file. If you build the producer application yourself, the configuration file is created and placed in the root for you when building.

Before running the producer to test loads, you must specify the following details in the configuration file.

Obtaining configuration details

Obtain the required configuration details as follows:

1. Log in to your Event Streams UI.
2. Click Connect to this cluster on the right.
3. Go to the Connect a client tab.
4. Locate the details:
   
   • For the bootstrap.servers, copy the address from the Bootstrap server section.
   • To downloaded the JKS keystore file, go to the Certificates section, and download the server certificate from the Java truststore section. Set the ssl.truststore.location to the full path and name of the downloaded file.
   • To generate API keys, go to the API key section and follow the instructions.

You can secure access to your topics as described in managing access.

Running

Create a load on your Event Streams Kafka cluster by running the es-producer.jar command. You can specify the load size based on the provided predefined values, or you can provide specific values for throughput and total messages to determine a custom load.

Using predefined loads

To use a predefined load size from the producer application, use the es-producer.jar with the -s option:

java -jar target/es-producer.jar -t <topic-name> -s <small/medium/large>

For example, to create a large message load based on the predefined large load size, run the command as follows:

java -jar target/es-producer.jar -t myTopic -s large

This example creates a large message load, where the producer attempts to send a total of 6,000,000 messages at a rate of 100,000 messages per second to the topic called myTopic.

The following table lists the predefined load sizes the producer application provides.

Using user-defined loads

You can generate a custom message load using your own settings.

For example, to test the load to the topic called myTopic with custom settings that create a total load of 60,000 messages with a size of 1024 bytes each, at a maximum throughput rate of 1000 messages per second, use the es-producer.jar command as follows:

java -jar target/es-producer.jar -t myTopic -T 1000 -n 60000 -r 1024

The following table lists all the parameter options for the es-producer.jar command.

Note: You can override the parameter values by using the environment variables listed in the following table. This is useful, for example, when using containerization, and you are unable to specify parameters on the command line.

Note: If you set the size using -s when running es-producer.jar, you can only override it if both the ES_NUM_RECORDS and ES_THROUGHPUT environment variables are set, or if ES_SIZE is set.