Handle evolving formats by using a schema registry

Scenario

The customer satisfaction team wants to respond to customers who cancel their orders in the moment, to attempt to convert some of these cancellations into new orders. For example, they are considering promptly reaching out to customers who cancel an order because they changed their mind, with voucher codes or information about similar products.

The format of events on this topic is expected to evolve over time, so the customer satisfaction team will connect Event Processing to the schema registry to continue to process without errors.

Before you begin

The instructions in this tutorial use the Tutorial environment, which includes a selection of topics each with a live stream of events, created to allow you to explore features in IBM Event Automation. Following the setup instructions to deploy the demo environment gives you a complete instance of IBM Event Automation that you can use to follow this tutorial for yourself.

Versions

This tutorial uses the following versions of Event Automation capabilities. Screenshots might differ from the current interface if you are using a newer version.

Event Streams 11.5.0
Event Processing 1.2.0

Instructions

Step 1 : Discover the topic to use

For this scenario, you need a source of order cancellation events.

Go to the Event Streams topics page.

If you need a reminder of how to access the Event Streams web UI, you can review Accessing the tutorial environment.
Find the CANCELLATIONS.REG topic.

Click into the topic to review the events that are available here.

Tip: Keep the Event Streams UI open. It is helpful to have the topic available while you start to work on your Event Processing flows, as you can refer to the connection information for the cluster as you work. Complete the following steps in a separate browser window or tab.

Go to the Event Streams schema registry page.
Find the CANCELLATIONS.REG-value schema.

Click into the schema to review the format of events on the CANCELLATIONS.REG topic.

Tip: Keep the Event Streams UI open. It is helpful to have the schema available while you start to work on your Event Processing flows, as you can refer to the event information as you work. Complete the following steps in a separate browser window or tab.

Step 2 : Create a flow

Go to the Event Processing home page.

If you need a reminder about how to access the Event Processing home page, you can review Accessing the tutorial environment.
Create a flow, and give it a name and description to explain that you will use it to create a stream of order cancellation events.

Step 3 : Provide a source of events

The next step is to bring the stream of events you discovered in the catalog into Event Processing.

Update the Event source node.

Hover over the node and click Edit to configure the node.
Add a new event source.

Click Next.
Get the server address for the topic from the Event Streams topic page.

As Event Streams is in the same cluster as Event Processing, you can use the internal listener for this.

Click the Copy icon next to the Internal listener address to copy the address to the clipboard.
Configure the new event source.

Paste the server address that you copied from Event Streams in the previous step, and click Next.
Copy the username and password from Event Endpoint Management and paste into Event Processing to allow access to the topic.

The username starts with eem-.

Click Next.

If you need a reminder of the password for the kafka-demo-apps user, you can review the Accessing Kafka topics.
Select the CANCELLATIONS.REG topic.
Click Next. The Message format pane is displayed.

The format Avro (schema registry) is auto-selected in the Message format drop-down.
Get the schema registry endpoint URL from Event Streams.

Click the Copy icon in the Schema registry endpoint section to copy the endpoint URL to the clipboard.
Paste the schema registry endpoint URL into the Schema registry URL box. Append /apis/ccompat/v6 to the end of the URL.
Use the same username and password for the kafka-demo-apps user for accessing the schema registry.
Click Next to auto-populate the Avro schema in the Avro schema field.

The information in the schema enables Event Processing to give guidance for creating Event Processing nodes.

Click Next.
In the Key and headers pane, click Next.

Note: The key and headers are displayed automatically if they are available in the selected topic message.
In the Event details pane, enter the node name as cancellations in the Node name field.
Set the lateness to 0.
Click Configure to finalize the event source.

Step 4 : Test the flow

The next step is to run your Event Processing flow and view the results.

Use the Run menu, and select Include historical to run your filter on the history of door badge events available on this Kafka topic.

Click Stop after you have reviewed the events.

Step 5 : Filter to events of interest

The next step is to identify cancellations where the customer changed their mind.

Create a Filter node and link it to the event source node.

Create a filter node by dragging one onto the canvas. You can find the filter node in the Processors section of the left panel.

Click and drag from the small gray dot on the event source to the matching dot on the filter node.

Hover over the node and click Edit to configure the node.
Give the filter node a name that describes the events it should identify: customer changed their mind.

Click Next.
Define a filter that matches cancellations events with a reason value that equals CHANGEDMIND.

Click Add to expression.
Click Configure to finalize the filter.

Step 6 : Test the flow

The final step is to run your Event Processing flow and view the results.

Use the Run menu, and select Include historical to run your filter on the history of cancellation events available on this Kafka topic.

Verify that all events are for cancellation events with a reason of CHANGEDMIND.

Recap

You used a schema registry in the event source node to dynamically provide schemas for a stream of events. This means that your Event Processing flow will continue to process if a compatible change is made to the schema for future events on the topic.