Using complex events with nested properties and arrays

Scenario

Using complex events with nested properties and arrays to identify a situation and extract only the necessary information. In this tutorial, you will see how to identify orders that contain out-of-stock products and extract the required information to handle these orders to notify customers who placed the orders.

Before you begin

The instructions in this tutorial use the Tutorial environment, which includes a selection of topics each with a live stream of events, created to allow you to explore features in IBM Event Automation. Following the setup instructions to deploy the demo environment gives you a complete instance of IBM Event Automation that you can use to follow this tutorial for yourself.

Versions

This tutorial uses the following versions of Event Automation capabilities. Screenshots can differ from the current interface if you are using a newer version.

Event Endpoint Management 11.3.0
Event Processing 1.2.0

Instructions

Step 1 : Discover the topics to use

For this scenario, you need a source for the Online orders events and the Out of stock notifications events.

Go to the Event Endpoint Management catalog.

If you need a reminder about how to access the Event Endpoint Management catalog, you can review Accessing the tutorial environment.
Find the Online orders and Out of stock notifications topics.
Click into the topics to review the information about the events that are available here. Look at the schema to see the properties in the events, and get an idea of what to expect from events on these topics.

Tip: Keep this page open. It is helpful to have the catalog available while you work on your event processing flows, as it allows you to refer to the documentation about the events as you work. Complete the following steps in a separate browser window or tab.

Step 2 : Create a flow

Go to the Event Processing home page.

If you need a reminder about how to access the Event Processing home page, you can review Accessing the tutorial environment.
Create a flow, and give it a name and description to explain that you will use it to create a stream of orders with out-of-stock products.

Step 3 : Provide a source of events

The next step is to bring the stream of events you discovered in the catalog into Event Processing.

Add an Event source node.

When you create a flow, an event source node is automatically added to your canvas. A purple checkbox is displayed on the event source node indicating that the node is yet to be configured.

Hover over the node and click Edit to configure the node.
Add a new event source.

Click Next.
Get the server address for the event source from the Event Endpoint Management catalog page.

Click the Copy icon next to the Servers address to copy the address to the clipboard.
Configure the new event source.

In the Server field, paste the server address that you copied from Event Endpoint Management in the previous step.

Click Next.
Generate access credentials for accessing this stream of events from the Event Endpoint Management page.

Click Generate access credentials at the top of the page, and provide your contact details.
Copy the username and password from Event Endpoint Management and paste into Event Processing to allow access to the topic.

The username starts with eem-.

Click Next.
Select the ORDERS.ONLINE topic to process events from, and click Next.

Click Next.
Get the schema for orders events from Event Endpoint Management.

Click the Copy icon in the schema section to copy the schema to the clipboard.

You need to give Event Processing a description of the events available from the topic. The information in the schema enables Event Processing to give guidance for creating event processing nodes.
The Avro message format is auto-selected in the Message format drop-down. Paste the schema into the Avro schema field, and click Next.
In the Key and headers pane, click Next.

Note: The key and headers are displayed automatically if they are available in the selected topic message.
In the Event details pane, enter the node a name that describes this stream of events: Orders.
Change the type of the ordertime property to Timestamp (with time zone).

Note: The ordertime string is converted to a timestamp to use ordertime as event time. Only properties with a timestamp type can be used as event time to perform time-based processing.
Configure the event source to use the ordertime property as the source of the event time, and to tolerate lateness of up to 1 minute.
Click Configure to finalize the event source.
Add a new Event source node to your canvas and repeat steps 1-10 under step 3 to add a source for Out of stock notifications events.
Configure the event source to use the outofstocktime property as the source of the event time, and to tolerate lateness of up to 1 minute.
Click Configure to finalize the event source.

Step 4 : Merge the two event streams

The next step is to define a join node to merge the two event streams to identify orders with out-of-stock products.

Add a Join node and link it to your event sources.

Create a join node by dragging one onto the canvas. You can find this in the Joins section of the left panel.

Click and drag from the small gray dot on the event source to the matching dot on the join node.

Hover over the node and click Edit to configure the node.
Give the join node a name that describes what it will do: Orders with out-of-stock products.

Click Next.
Define the join condition to match the product descriptions in an order with the product in out-of-stock events.
```
ARRAY_CONTAINS(products, CONCAT_WS ( ' ', product.size, product.material, product.style, product.name))
```
Click Next.
To identify products that have run out of stock within one hour of being ordered, use the Out-of-stock(outofstocktime) event as the event to detect and Orders(ordertime) event as the event to set the time window with a one hour time window.

Click Next.
Rename the property orders.id to orderid to remove the ambiguity with the outofstock.id property.
Remove the outofstock.id property to exclude it from the merged events.
Click Configure to finalize the join.

Step 5 : Test the flow

The next step is to run your event processing flow and view the results.

Use the Run menu, and select Include historical to run your filter on the history of orders and out-of-stock products.

Verify that the orders with out-of-stock products are being correctly displayed.

Step 6 : Derive additional properties

After identifying orders with out-of-stock products, we need to extract the necessary information to handle these orders and notify customers who placed them.

The next step is to define transformations that will derive additional properties to add to the events.

Add a Transform node and link it to your Join node: Orders with out-of-stock products.

Create a transform node by dragging one onto the canvas. You can find this in the Processors section of the left panel.

Click and drag from the small gray dot on the join node to the matching dot on the transform node.

Hover over the node and click Edit to configure the node.
Give the transform node a name that describes what it will do: Extract contact and product information.

Click Next.

Create the following new additional properties using the transform node.

Property name	Expression	Description
contactname	customer.name	Creates a contactname property and sets it to the customer name.
contactemail	customer.emails[1]	Creates a contactemail property and sets it to the first email in the array of emails provided for the customer. Note: The index of arrays starts at 1, not at 0.
contactphone	IF ( address.billingaddress.phones IS NOT NULL, address.billingaddress.phones[1], address.shippingaddress.phones[1] )	Creates a contactphone property and sets it to the first phone number in the billing address, else set it to the first phone number in the shipping address. Note: Accessing an array element with an index outside the array data range returns null. For example, phones[1] return null if the array is empty or null.
billingaddress	address.billingaddress	Creates a billingaddress property and sets it to extract only the billing address information.
outofstockproduct	CONCAT_WS(‘ ‘, product.size, product.material, product.style, product.name)	Creates an outofstockproduct property and sets it to the product description, which is retrieved by concatenating the size, material, style, and name properties of the product separated by a space character.

Click Next.

After extracting all the required information, remove the following properties which are no longer necessary:
- customer
- address
- product
Click Configure to finalize the transform.

Step 7 : Test the flow

The final step is to run your event processing flow and view the results.

Use the Run menu, and select Include historical to run your filter on the history of orders and out-of-stock products.

Verify that the output events now only contain the contact and product information.

Recap

You used a join node to merge the orders and out-of-stock events to identify out-of-stock products. Then, you used a transform node to add more properties and extract necessary information from complex objects and arrays.