Creating and testing message loads

Event Streams provides a high-throughput producer application you can use as a workload generator to test message loads and help validate the performance capabilities of your cluster.

You can use one of the predefined load sizes, or you can specify your own settings to test throughput. Then use the test results to ensure your cluster setup is appropriate for your requirements, or make changes as needed, for example, by changing your scaling settings.

Downloading

You can download the latest pre-built producer application.

Alternatively, you can clone the project from GitHub. However, if you clone from GitHub, you have to build the producer.

Building

If you cloned the Git repository, build the producer as follows:

1. Ensure you have Java version 8. Also, ensure you have Apache Maven installed on your system.
2. Ensure you have cloned the Git project.
3. Open a terminal and change to the root directory of the event-streams-sample-producer project.

4. Run the following command:

   mvn install
   

   You can also specify your root directory using the -f option as follows:

   mvn install -f <path_to>/pom.xml
   

5. The es-producer.jar file is created in the /target directory.

Configuring

The producer application requires configuration settings that you can set in the provided producer.config template configuration file.

Note: The producer.config file is located in the root directory. If you downloaded the pre-built producer, you have to run the es-producer.jar with the -g option to generate the configuration file. If you build the producer application yourself, the configuration file is created and placed in the root for you when building.

Before running the producer to test loads, you must specify the bootstrap.servers and any required security configuration details in the configuration file.

Obtaining configuration details

The bootstrap servers address can be obtained from the Event Streams UI as described in the following steps. Other methods to obtain the bootstrap servers address are described in connecting client.

1. Log in to the Event Streams UI from a supported web browser (see how to determine the login URL for your Event Streams UI).
2. Click Connect to this cluster on the right.
3. Click the Resources tab.
4. Go to the Kafka listener and credentials section.
5. Copy the address from one of the External listeners.

The producer application might require credentials for the listener chosen in the previous step. For more information about these credentials, see the information about managing access.

Obtain the required credentials as follows:

1. Log in to the Event Streams UI from a supported web browser (see how to determine the login URL for your Event Streams UI).
2. Click Connect to this cluster on the right.
3. Go to the Resources tab.
4. Scroll down to the Kafka listener and credentials section.
5. Click the button next to the listener chosen as the bootstrap.servers configuration. If present, the button will either be labeled Generate SCRAM credentials or Generate TLS credentials.
6. Select Produce messages, consume messages and create topics and schemas and click Next.
7. Select A specific topic, enter the name of a topic to produce to and click Next.
8. Select All consumer groups and click Next.
9. Select No transactional IDs and click Generate credentials.
10. Retrieve the generated credentials:
    
    • If using SCRAM note down the Username and password.
    • If using TLS click Download certificates and extract the contents of the resulting .zip file to a preferred location.

Obtain the Event Streams certificate as follows:

1. Log in to the Event Streams UI from a supported web browser (see how to determine the login URL for your Event Streams UI).
2. Click Connect to this cluster on the right.
3. Go to the Resources tab.
4. Scroll down to the Certificates section.
5. In the PKCS12 certificate section click Download certificate.
6. Note down the generated password displayed in the Certificate password section.

Updating the configuration file

Before updating the file, obtain the credentials by following the steps in Obtaining configuration details.

Update the producer.config file with your configuration details using the following table as guidance.

Running

Create a load on your Event Streams Kafka cluster by running the es-producer.jar command. You can specify the load size based on the provided predefined values, or you can provide specific values for throughput and total messages to determine a custom load.

Using predefined loads

To use a predefined load size from the producer application, use the es-producer.jar with the -s option:

java -jar target/es-producer.jar -t <topic-name> -s <small/medium/large>

For example, to create a large message load based on the predefined large load size, run the command as follows:

java -jar target/es-producer.jar -t my-topic -s large

This example creates a large message load, where the producer attempts to send a total of 6,000,000 messages at a rate of 100,000 messages per second to the topic called my-topic.

The following table lists the predefined load sizes the producer application provides.

Specifying a load

You can generate a custom message load using your own settings.

For example, to test the load to the topic called my-topic with custom settings that create a total load of 60,000 messages with a size of 1024 bytes each, at a maximum throughput rate of 1000 messages per second, use the es-producer.jar command as follows:

java -jar target/es-producer.jar -t my-topic -T 1000 -n 60000 -r 1024

The following table lists all the parameter options for the es-producer.jar command.

Note: You can override the parameter values by using the environment variables listed in the following table. This is useful, for example, when using containerization, and you are unable to specify parameters on the command line.

Note: If you set the size using -s when running es-producer.jar, you can only override it if both the ES_NUM_RECORDS and ES_THROUGHPUT environment variables are set, or if ES_SIZE is set.