Read about the key concepts and terms used in Event Endpoint Management. The following diagram demonstrates how the key concepts work together.
Topic
A Kafka topic, which contains a set of related events.
Event source
An event source is the term used to describe a topic when it has been added to Event Endpoint Management. A Kafka administrator can make event sources accessible by publishing an option to the catalog where it becomes an event endpoint. Application developers can discover and browse the available event endpoints and decide which ones they want access to. They can then configure their applications to either consume from the event endpoint or produce to it.
Option
A Kafka administrator can create an option for an event source that enables them to control how the topic’s stream of events are made available within the catalog. When an option is published to the catalog, it is called an event endpoint.
Controls
You can add controls to options so that you have greater management over how event source data is presented to applications.
You can add the following controls to an option on a consume-enabled event source:
- Approval: Use the approval control to force users to request access to your event endpoint.
- Redaction: Use a redaction control to hide sensitive data.
- Schema filtering: Use the schema filtering control to manage how the data in your event source is presented to consumers and to ensure consistency.
You can add the following controls to an option on a produce-enabled event source:
- Approval: Use the approval control to force users to request access to your event endpoint.
- Schema enforcement: Use the schema enforcement control to enforce the shape of data that is allowed to be produced to your event endpoint.
Event endpoint
When a Kafka administrator publishes an option to the Event Endpoint Management catalog, the option exposes an event source as an event endpoint. This contains all the descriptive elements provided when describing a topic as an event source, and the management configuration specified in the published option. If other options have already been published relating to an event source, they are available under the event endpoint in the catalog.
For an application developer, the event endpoint is the object that they can discover in the catalog, and behaves like a topic when it is accessed through the Event Gateway.
Catalog
The catalog lists all available event endpoints and, if applicable, their available options. Application developers in your organization can use the catalog to browse the available event endpoints, and to view more information about each of them, including a description, tags, sample messages, schema details if used, and so on, enabling access to the stream of events represented by the event source using the published options.
Kafka administrators can use the catalog to check what options are published and made available to others in the organization.
Cluster
Kafka runs as a cluster of one or more servers (Kafka brokers). The load is balanced across the cluster by distributing it amongst the servers.
Subscription
Application developers configure their applications to subscribe to the stream of events, providing access to the message content from the event stream, and generating the required credentials for their application to consume from the event endpoint.
Kafka administrators can limit subscriptions by adding the approval control to the options of an event source and they can manage subscriptions that are created for their event sources in the Event Endpoint Management Topic Detail page.
Consume
An application developer can discover event endpoints that are published to the catalog. The application developer must subscribe to an option on a consume-enabled event source to get access for a client application to consume Kafka events from the related topic through the Event Gateway, by using standard Kafka client libraries.
Produce
An application developer can write events to an event endpoint that is published to the catalog. The application developer must subscribe to an option on a produce-enabled event source to get access for a client application to produce Kafka events to the related topic through the Event Gateway, by using standard Kafka client libraries.
Event Manager
The Event Manager component adopts your Kafka topics into a central location after you describe them as event sources, and you can then generate event endpoints for sharing with others. The Event Manager provides features to configure how you want to present the topic data by using event endpoints, which can be accessed through the Event Gateway.
Event Manager instances are defined by the EventEndpointManagement custom resource type.
Event Gateway
Access to event sources are managed by the Event Gateway. The Event Gateway handles the incoming requests from applications to access an event source, routing traffic securely between the Kafka cluster and the application.
The Event Gateway is independent of your Kafka clusters, making access to topics possible without requiring any changes to your Kafka cluster configuration.
Event Gateway instances are defined by the EventGateway custom resource type.
Event Gateway group
An Event Gateway group is a group of Event Gateway instances that can be logically grouped and scaled together. When deploying an Event Gateway, you can define gateway groups by name, and any Event Gateway instances with the same group name are considered a member of that group.
The following are examples of Event Gateway groupings:
- Event Gateways that are deployed to manage Kafka traffic to designated Kafka clusters.
- Event Gateways that are colocated within the same geography.
- Event Gateways that represent different Kafka topics and streams at different stages of adoptable maturity, such as development, test, or production.
When Event Gateways start, they join their defined Event Gateway group and pull any configuration that is specified for that group. The configuration for an Event Gateway group is determined when publishing an event source, where you can select which group the event source must be accessible through. By publishing an event source to an Event Gateway group, you can restrict which gateways have access to an event source without specifying the gateway instances directly. Event Gateway instances can be added or removed from a group at any time, providing flexibility as the use of event sources matures.
Note: A gateway instance can only be a member of one gateway group.
Important: When removing a gateway deployment, work with subscribed users to ensure their clients are configured to work with another member of the same gateway group, so they can continue to consume your event source.
Message
The unit of data in Kafka. Each message is represented as a record, which comprises two parts: key and value. The key is commonly used for data about the message and the value is the body of the message. Message is also sometimes referred to as event data and record.
To learn more about key concepts, see the Apache Kafka documentation.