When processing events we can modify events to remove some properties from the events. Transforms work on individual events in the stream.
Scenario: Redact personal information from order events
The operations team wants to enable another team to analyze order events, however this other team is not permitted access to personally-identifiable information (PII) about customers. They need to be able to process order events without customer PII.
Before you begin
The instructions in this tutorial use the Tutorial environment, which includes a selection of topics each with a live stream of events, created to allow you to explore features in IBM Event Automation. Following the setup instructions to deploy the demo environment gives you a complete instance of IBM Event Automation that you can use to follow this tutorial for yourself.
This tutorial was written using the following versions of Event Automation operators. Screenshots may differ from the current interface if you are using a newer version.
- Event Streams 3.3.0
- Event Endpoint Management 11.1.1
- Event Processing 1.1.1
Step 1 : Create a flow
Go to the Event Processing home page.
If you need a reminder of how to access the Event Processing home page, you can review Accessing the tutorial environment.
Create a flow, and give it a name and description to explain that you will use it to redact customer information from orders.
Step 2 : Provide a source of events
The next step is to bring the stream of events to process into the flow. We will reuse the topic connection information from an earlier tutorial.
Create an Event source node.
Create an event source node by dragging one onto the canvas. You can find this in the Events section of the left panel.
ORDERStopic that you used in the Identify orders from a specific region tutorial.
Tip: If you haven’t followed that tutorial, you can click Add new event source instead, and follow the Provide a source of events steps in the previous tutorial to define a new Event source from scratch.
The schema for events on this topic defined before is displayed. Click Configure.
To rename the event source, click the Edit icon , and enter a name for your event source node in the Details > Node name section.
Step 3 : Define the transformation
The next step is to define the transformation that will remove the customer personal information from events.
Add a Transform node and link it to your event source.
Create a transform node by dragging one onto the canvas. You can find this in the Processors section of the left panel.
Click and drag from the small gray dot on the event source to the matching dot on the transform node.
Did you know? Instead of dragging the node, you can add a node onto the canvas and automatically connect it to the last added node by double-clicking a node within the palette. For example, after configuring an event source node, double-click any processor node to add and connect the processor node to your previously configured event source node.
Give the transform node a name that describes what it will do:
redact personal info.
Hover over the node and click Edit to configure the node.
As you aren’t creating any new properties, click Next.
(You can learn about creating custom properties in the next tutorial.)
customeridproperties from the events.
Remove properties from events by clicking the remove property icon next to their name.
Step 4 : Testing the flow
The final step is to run your event processing flow and view the results.
Use the Run menu, and select Include historical to run your filter on the history of order events available on this Kafka topic.
Click the redact transform node to see a live view of results from your transform. It is updated as new events are emitted onto the orders topic.
You used a transform node to redact sensitive data from order events.
In the next tutorial, you will try a windowed operation to process events that occur within a time window.