1 - Filter events based on particular properties

Filter

When processing events, we can use filter operations to select a subset that we want to use. Filtering works on individual events in the stream.

Scenario : Identify orders from a specific region

The EMEA operations team wants to move away from reviewing quarterly sales reports and be able to review orders in their region as they occur.

Identifying large orders as they occur will help the team identify changes that are needed in sales forecasts much earlier. These results can also be fed back into their manufacturing cycle so they can better respond to demand.

Before you begin

The instructions in this tutorial use the Tutorial environment, which includes a selection of topics each with a live stream of events, created to allow you to explore features in IBM Event Automation. Following the setup instructions to deploy the demo environment gives you a complete instance of IBM Event Automation that you can use to follow this tutorial for yourself.

Versions

This tutorial uses the following versions of Event Automation capabilities. Screenshots may differ from the current interface if you are using a newer version.

Event Streams 11.5.0
Event Endpoint Management 11.3.0
Event Processing 1.2.0

Instructions

Step 1 : Discover the topic to use

For this scenario, you need a source of order events. A good place to discover sources of event streams to process is the catalog in Event Endpoint Management.

Go to the Event Endpoint Management catalog.

If you need a reminder of how to access the Event Endpoint Management catalog you can review Accessing the tutorial environment.

If there are no topics in the catalog, you may need to complete the tutorial setup step to populate the catalog.
Find the Orders topic.
Click into the topic to review the information about the events that are available here.

Look at the schema to see the properties in the order events, and get an idea of what to expect from events on this topic.

Tip: Keep this page open. It is helpful to have the catalog available while you work on your event processing flows, as it allows you to refer to the documentation about the events as you work. Do the following steps in a separate browser window or tab.

Step 2 : Create a flow

The next step is to start processing this stream of events, to create a custom subset that contains the events that you are interested in.

Go to the Event Processing home page.

If you need a reminder of how to access the Event Processing home page, you can review Accessing the tutorial environment.
Create a flow, and give it a name and description to explain that you will use it to identify orders made in the EMEA region.

Step 3 : Provide a source of events

The next step is to bring the stream of events you discovered in the catalog into Event Processing.

Update the Event source node.

Hover over the node and click Edit to configure the node.
Add a new event source.
Get the server address for the event source from the Event Endpoint Management topic page.

Click the Copy icon next to the Servers field to copy the address to the clipboard.
Configure the new event source.

In Event Processing, paste the server address in the Server field that you copied from Event Endpoint Management in the previous step.
Generate access credentials for accessing this stream of events from the Event Endpoint Management page.

Click Generate access credentials at the top of the page, and provide your contact details.

Did you know? Providing your contact details allows the owner of the topic to know who is accessing their stream of events.
Copy the username and password from Event Endpoint Management and paste into Event Processing to allow access to the topic.

The username starts with eem-.

Did you know? The username and password you created is unique to you, and is only for accessing this topic. If you need to revoke this password, you can do it without impacting other users of this topic.

Click Next.
Select the ORDERS topic to process events from, and click Next.
Get the schema for order events from Event Endpoint Management.

Switch to the JSON schema view, and then click the Copy icon for the schema to copy it to the clipboard.

You need to give Event Processing a description of the events available from the topic. The schema will enable Event Processing to give guidance for creating event processing nodes.
The Avro message format is auto-selected in the Message format drop-down. Paste the schema into the Avro schema field, and click Next.
In the Key and headers pane, click Next.

Note: The key and headers are displayed automatically if they are available in the selected topic message.
In the Event details pane, enter the node a name that describes this stream of events: Orders.

Note: The auto-detected key and headers are displayed unselected in the Event details pane under the Property name column.

Leave the event source to be saved for later reuse.

Tip: Saving the connection details makes the later steps in the tutorial that use this same topic quicker. It avoids you needing to enter these details again.
Click Configure to finalize the event source.

Step 4 : Define the filter

The next step is to start processing this stream of events, by creating the filter that will select the custom subset with the events that you are interested in.

Create a Filter node and link it to your event source.

Create a filter node by dragging one onto the canvas. You can find this in the Processors section of the left panel.

Click and drag from the small gray dot on the event source to the matching dot on the filter node.

Did you know? Instead of dragging the node, you can add a node onto the canvas and automatically connect it to the last added node by double-clicking a node within the palette. For example, after configuring an event source node, double-click any processor node to add and connect the processor node to your previously configured event source node.
Give the filter node a name that describes the events it should identify: EMEA orders.

Hover over the node and click Edit to configure the node.
Use the assistant to define a filter that matches events with a region value of EMEA.
Click Add to expression.
Click Configure to finalize the filter.

Step 5 : Test the flow

The next step is to run your event processing flow and view the results.

Use the Run menu, and select Include historical to run your filter on the history of order events available on this Kafka topic.

Did you know? “Include historical” is useful while you are developing your flows, as it means that you don’t need to wait for new events to be produced to the Kafka topic. You can use all of the events already on the topic to check that your flow is working the way that you want.
A live view of results from your filter is updated as new events are emitted onto the orders topic.
When you have finished reviewing the results, you can stop this flow.

Recap

You created your first event processing flow.

You have seen how to discover and request access to a topic in the catalog, and register it as a source of events for processing.

You used a filter node to specify a subset of events on the topic that you are interested in.

Next step

In the next tutorial, you will try another stateless operation to learn how to modify events in the stream.