If you have IBM MQ or another service running on IBM Cloud Private, you can use Kafka Connect and one or more connectors to flow data between your instance of IBM Event Streams and the service on IBM Cloud Private. In this scenario it makes sense to run Kafka Connect in IBM Cloud Private as well.
The connector catalog contains a list of connectors that have been verified with Event Streams. Go to the connector catalog and download the JAR file(s) for any connectors you want to use.
The JAR files for the IBM MQ source and sink connectors can be downloaded from the IBM Event Streams UI. Log in to your IBM Event Streams UI, click the Toolbox tab and look for the tile called “Add connectors to your Kafka Connect environment”.
Building a Kafka Connect Docker image
The Event Streams UI provides a toolbox page to help you get started with Kafka Connect. This provides a Dockerfile that builds a custom Kafka Connect with the Connectors you include.
If you do not already have the Dockerfile, follow the steps to download the Kafka Connect ZIP and build a Docker image.
- In the Event Streams UI, click the Toolbox tab. Scroll to the Connectors section.
- Go to the Set up a Kafka Connect environment tile, and click Set up.
- If you have not already done so follow the instructions to create three topics for Kafka Connect to use.
- You need to provide an API key for Kafka Connect that has permission to Produce, Consume and create Topics. Paste in your API key, or click the button to generate one.
NOTE: You must have a cluster admin role to generate an API key.
- Click Download Kafka Connect ZIP to download the zip.
- Extract the contents of the Kafka Connect
.zipfile to a local directory.
- Copy the connector JAR files you downloaded earlier into the
connectorsfolder in the extracted
cp <path_to_your_connector>.jar <extracted_zip>/connectors
- Build the container:
docker build -t kafkaconnect:0.0.1 .
Uploading the Kafka Connect container
To make the Kafka Connect container available on IBM Cloud Private it needs to be pushed to your IBM Cloud Private container registry.
- Set up your Kubernetes command-line tool
kubectlto access your IBM Cloud Private instance, for example, by running
- Create a namespace to deploy the Kafka Connect workers to:
kubectl create namespace <namespace>
- Log in to the Docker private image registry:
cloudctl login -a https://<cluster_CA_domain>:8443 docker login <cluster_CA_domain>:8500
For more information, see the IBM Cloud Private documentation.
- Retag and push the Docker image as follows:
docker tag kafkaconnect:0.0.1 <cluster_CA_domain>:8500/<namespace>/kafkaconnect:0.0.1 docker push <cluster_CA_domain>:8500/<namespace>/kafkaconnect:0.0.1
- Check this has worked by logging into your IBM Cloud Private UI and clicking on Container Images in the menu.
Note: The namespace you provide is the one you will run the Kafka Connect workers in.
Creating a Secret resource for the Kafka Connect configuration
To enable updates to the Kafka Connect configuration the running container will need access to a Kubernetes resource containing the contents of connect-distributed.properties. The file is included in the extracted ZIP for Kafka Connect from the Event Streams UI. This file includes API keys so create a Secret:
kubectl -n <namespace> create secret generic connect-distributed-config --from-file=<extracted_zip>/config/connect-distributed.properties
Creating a ConfigMap resource for the Kafka Connect log4j configuration
To enable updates to the Kafka Connect logging configuration create a ConfigMap with the contents of connect-log4j.properties. The file is included in the extracted ZIP for Kafka Connect from the Event Streams UI:
kubectl -n <namespace> create configmap connect-log4j-config --from-file=<extracted_zip>/config/connect-log4j.properties
Creating the Kafka Connect deployment
To create the Kafka Connect deployment first create a yaml file called
kafka-connect.yaml with the following contents: (Replace
<namespace> with your IBM Cloud Private namespace)
# Deployment apiVersion: apps/v1 kind: Deployment metadata: name: kafkaconnect-deploy labels: app: kafkaconnect spec: replicas: 1 selector: matchLabels: app: kafkaconnect template: metadata: namespace: <namespace> labels: app: kafkaconnect spec: securityContext: runAsNonRoot: true runAsUser: 5000 containers: - name: kafkaconnect-container image: kafkaconnect:0.0.1 readinessProbe: httpGet: path: / port: 8083 livenessProbe: httpGet: path: / port: 8083 ports: - containerPort: 8083 volumeMounts: - name: connect-config mountPath: /opt/kafka/config/connect-distributed.properties subPath: connect-distributed.properties - name: connect-log4j mountPath: /opt/kafka/config/connect-log4j.properties subPath: connect-log4j.properties volumes: - name: connect-config secret: secretName: connect-distributed-config - name: connect-log4j configMap: name: connect-log4j-config --- # Service apiVersion: v1 kind: Service metadata: name: kafkaconnect-service labels: app: kafkaconnect-service spec: type: NodePort ports: - name: kafkaconnect protocol: TCP port: 8083 selector: app: kafkaconnect
This defines the deployment that will run Kafka Connect and the service used to access it.
Create the deployment and service using:
kubectl -n <namespace> apply -f kafka-connect.yaml
kubectl -n <namespace> get service kafkaconnect-service to view your running services. The port mapping shows
8083 being mapped to an external port. Use the external port to verify the IBM MQ Connectors you included have been installed:
Running a connector
To start a Connector instance, you need to create a JSON file with the connector configuration. Most connectors will have an example in their documentation. For the IBM MQ connectors this file can be generated in the Event Streams UI or CLI. See connecting to IBM MQ for more details.
Once you have a JSON file use the
/connectors endpoint to start the connector:
curl -X POST http://<serviceIP>:<servicePort>/connectors -H "Content-Type: application/json" -d @mq-source.json
For more information about the other REST API endpoints (such as pausing, restarting, and deleting connectors) see the Kafka Connect REST API documentation.