Introduction

Event Streams is an event-streaming platform based on the Apache Kafka® project and incorporates the open-source Strimzi technology.

Event Streams uses Strimzi to deploy Apache Kafka in a resilient and manageable way, and provides a range of additional capabilities to extend the core functionality.

Event Streams architecture

Event Streams features include:

  • Apache Kafka deployed by Strimzi that distributes Kafka brokers across the nodes of a Kubernetes cluster, creating a highly-available and resilient deployment.

  • An administrative user interface (UI) focused on providing essential features for managing production clusters, in addition to enabling developers to get started with Apache Kafka. The Event Streams UI facilitates topic lifecycle control, message filtering and browsing, client metric analysis, schema and geo-replication management, connection details and credentials generation, and a range of tools, including a sample starter and producer application.

  • A command-line interface (CLI) to enable manual and scripted cluster management. For example, you can use the Event Streams CLI to inspect brokers, manage topics, deploy schemas, and manage geo-replication.

  • Geo-replication of topics between clusters to enable disaster recovery.

  • A REST API for producing messages to Event Streams topics, expanding event source possibilities.

  • A schema registry to support the definition and enforcement of message formats between producers and consumers. Event Streams includes the open-source Apicurio Registry for managing schemas.

  • Health check information to help identify issues with your clusters and brokers.

  • Secure by default production cluster templates with authentication, authorization and encryption included, and optional configurations to simplify proof of concept or lite development environments.

  • Granular security configuration through Kafka access control lists to configure authorization and quotas for connecting applications.

Operators

Kubernetes Operators are designed to simplify the deployment and maintenance of complex applications. They do this by packaging up and abstracting away domain-specific knowledge of how to deploy, configure and operate the application and its component parts. An Operator can then extend Kubernetes by providing new Kubernetes constructs to support these abstractions, simplifying the initial deployment and subsequent lifecycle management of the application.

Strimzi uses Operators in this manner to facilitate the deployment of Kafka clusters. Event Streams builds on top of Strimzi to deploy not only the Kafka components but also the additional features outlined earlier. A new Kubernetes resource EventStreams is provided (in Kubernetes referred to as a kind), to allow the definition of a complete deployment that brings together the components provided by Strimzi and Event Streams. This information is defined in a standard YAML document and deployed in a single operation.

Operators provided by Event Streams

The following diagram shows the operators involved in an Event Streams deployment along with the resources they manage. Event Streams builds on the Strimzi core, and adds additional components to extend the base capabilities.

EventStreams Operator diagram.

The EventStreams Cluster Operator

The Strimzi Cluster Operator manages the deployment of core components within the Kafka cluster such as Kafka and ZooKeeper nodes.

The EventStreams Cluster Operator extends the Strimzi Cluster Operator to provision the additional components provided by Event Streams alongside these core components. A new custom resource Type called EventStreams is provided to manage this overall deployment.

The Entity Operator

Kafka provides authentication and authorization through Users and Access Control Lists. These define the operations that users can perform on specific resources within the cluster.

The User operator provides a new custom resource type called KafkaUser to manage these Kafka users and associated Access Control Lists and is a mandatory component of an Event Streams deployment.

Kafka topics are a resource within the cluster to which a series of records can be produced.

The optional Topic operator provides a new custom resource type called KafkaTopic to manage these Kafka topics. By default, the Topic operator is not part of the Event Streams cluster. Instead, to create and manage Kafka topics, use the Event Streams UI, CLI, and REST API provided mechanisms.

If there is a need to have Kafka topics represented as Kubernetes resources, the optional Topic operator can be included in the EventStreams definition, and will then be deployed as a container alongside the User operator within the Entity operator Pod.