Connecting to IBM MQ

You can set up connections between IBM MQ and Apache Kafka or Event Streams systems.

Available connectors

Connectors are available for copying data in both directions.

Kafka Connect: MQ source and sink connectors

Important: If you want to use IBM MQ connectors on IBM z/OS, you must prepare your setup first.

When to use

Many organizations use both IBM MQ and Apache Kafka for their messaging needs. Although they’re generally used to solve different kinds of messaging problems, users often want to connect them together for various reasons. For example, IBM MQ can be integrated with systems of record while Apache Kafka is commonly used for streaming events from web applications. The ability to connect the two systems together enables scenarios in which these two environments intersect.

Note: You can use an existing IBM MQ or Kafka installation, either locally or on the cloud. For convenience, it is recommended to run the Kafka Connect worker on the same Kubernetes cluster as Event Streams. If the network latency between MQ and Event Streams is significant, you might prefer to run the Kafka Connect worker close to the queue manager to minimize the effect of network latency. For example, if you have a queue manager in your datacenter and Kafka in the cloud, it’s best to run the Kafka Connect worker in your datacenter.

Configuration options

Find out the available configuration options for the IBM MQ source connector and the IBM MQ sink connector.

IBM MQ source connector

The configuration options for the Kafka Connect source connector for IBM MQ are as follows:

Name Description Type Default Valid values
topic The name of the target Kafka topic. string   Topic name
mq.queue.manager The name of the MQ queue manager. string   MQ queue manager name
mq.connection.mode The connection mode - bindings or client. string client client, bindings
mq.connection.name.list List of connection names for queue manager. string   host(port)[,host(port),…]
mq.channel.name The name of the server-connection channel. string   MQ channel name
mq.queue The name of the source MQ queue. string   MQ queue name
mq.exactly.once.state.queue The name of the MQ queue used to store state when running with exactly-once semantics. string   MQ state queue name
mq.user.name The user name for authenticating with the queue manager. string   User name
mq.password The password for authenticating with the queue manager. string   Password
mq.user.authentication.mqcsp Whether to use MQ connection security parameters (MQCSP). boolean true  
mq.ccdt.url The URL for the CCDT file containing MQ connection details. string   URL for obtaining a CCDT file
mq.record.builder The class used to build the Kafka Connect record. string   Class implementing RecordBuilder
mq.message.body.jms Whether to interpret the message body as a JMS message type. boolean false  
mq.record.builder.key.header The JMS message header to use as the Kafka record key. string   JMSMessageID, JMSCorrelationID, JMSCorrelationIDAsBytes, JMSDestination
mq.jms.properties.copy.to.kafka.headers Whether to copy JMS message properties to Kafka headers. boolean false  
mq.ssl.cipher.suite The name of the cipher suite for TLS (SSL) connection. string   Blank or valid cipher suite
mq.ssl.peer.name The distinguished name pattern of the TLS (SSL) peer. string   Blank or DN pattern
mq.ssl.keystore.location The path to the JKS keystore to use for SSL (TLS) connections. string JVM keystore Local path to a JKS file
mq.ssl.keystore.password The password of the JKS keystore to use for SSL (TLS) connections. string    
mq.ssl.truststore.location The path to the JKS truststore to use for SSL (TLS) connections. string JVM truststore Local path to a JKS file
mq.ssl.truststore.password The password of the JKS truststore to use for SSL (TLS) connections. string    
mq.ssl.use.ibm.cipher.mappings Whether to set system property to control use of IBM cipher mappings. boolean    
mq.batch.size The maximum number of messages in a batch (unit of work). integer 250 1 or greater
mq.message.mqmd.read Whether to enable reading of all MQMD fields. boolean false  
mq.max.poll.blocked.time.ms How long the connector will wait for the previous batch of messages to be delivered to Kafka before starting a new poll. integer 2000 It is important that this is less than the time defined for task.shutdown.graceful.timeout.ms as that is how long connect will wait for the task to perform lifecycle operations.
mq.client.reconnect.options Options governing MQ reconnection. string ASDEF ASDEF, ANY, QMGR, DISABLED
mq.message.receive.timeout The timeout (in milliseconds) for receiving messages from the queue manager before returning to Kafka Connect. long 2000 1 or greater
mq.receive.subsequent.timeout.ms The timeout (in milliseconds) for receiving messages from the queue manager on the subsequent receives before returning to Kafka Connect. long 0 1 or greater
mq.reconnect.delay.min.ms The minimum delay (in milliseconds) for reconnecting to the queue manager after a connection error. long 64 1 or greater
mq.reconnect.delay.max.ms The maximum delay (in milliseconds) for reconnecting to the queue manager after a connection error. long 8192 1 or greater
mq.receive.max.poll.time.ms Maximum time (in milliseconds) to poll messages in a single Kafka Connect task cycle. If set to 0, polling continues until batch size or a receive returns null. long 0 0 or greater
errors.deadletterqueue.topic.name The name of the Kafka topic to use as the dead letter queue (DLQ) for poison messages that fail during processing within the record builder component of the connector. string   If left blank (default), failed messages will not be written to a DLQ.
errors.deadletterqueue.context.headers.enable Whether to add error context headers to messages written to the DLQ. boolean false When enabled, additional headers describing the error will be included with each DLQ record.

IBM MQ sink connector

The configuration options for the Kafka Connect sink connector for IBM MQ are as follows:

Name Description Type Default Valid values
topics or topics.regex List of Kafka source topics. string   topic1[,topic2,…]
mq.queue.manager The name of the MQ queue manager. string   MQ queue manager name
mq.connection.mode The connection mode - bindings or client. string client client, bindings
mq.connection.name.list List of connection names for queue manager. string   host(port)[,host(port),…]
mq.channel.name The name of the server-connection channel. string   MQ channel name
mq.queue The name of the target MQ queue. string   MQ queue name
mq.exactly.once.state.queue The name of the MQ queue used to store state when running with exactly-once semantics. string   MQ state queue name
mq.user.name The user name for authenticating with the queue manager. string   User name
mq.password The password for authenticating with the queue manager string   Password
mq.user.authentication.mqcsp Whether to use MQ connection security parameters (MQCSP). boolean true  
mq.ccdt.url The URL for the CCDT file containing MQ connection details. string   URL for obtaining a CCDT file
mq.message.builder The class used to build the MQ message. string   Class implementing MessageBuilder
mq.message.body.jms Whether to generate the message body as a JMS message type. boolean false  
mq.time.to.live Time-to-live in milliseconds for messages sent to MQ. long 0 (unlimited) [0,…]
mq.persistent Send persistent or non-persistent messages to MQ. boolean true  
mq.ssl.cipher.suite The name of the cipher suite for TLS (SSL) connection. string   Blank or valid cipher suite
mq.ssl.peer.name The distinguished name pattern of the TLS (SSL) peer. string   Blank or DN pattern
mq.ssl.keystore.location The path to the JKS keystore to use for SSL (TLS) connections. string JVM keystore Local path to a JKS file
mq.ssl.keystore.password The password of the JKS keystore to use for SSL (TLS) connections. string    
mq.ssl.truststore.location The path to the JKS truststore to use for SSL (TLS) connections. string JVM truststore Local path to a JKS file
mq.ssl.truststore.password The password of the JKS truststore to use for SSL (TLS) connections. string    
mq.ssl.use.ibm.cipher.mappings Whether to set system property to control use of IBM cipher mappings. boolean    
mq.message.builder.key.header The JMS message header to set from the Kafka record key. string   JMSCorrelationID
mq.kafka.headers.copy.to.jms.properties Whether to copy Kafka headers to JMS message properties. boolean false  
mq.message.builder.value.converter The class and prefix for message builder’s value converter. string   Class implementing Converter
mq.message.builder.topic.property The JMS message property to set from the Kafka topic. string   Blank or valid JMS property name
mq.message.builder.partition.property The JMS message property to set from the Kafka partition. string   Blank or valid JMS property name
mq.message.builder.offset.property The JMS message property to set from the Kafka offset. string   Blank or valid JMS property name
mq.reply.queue The name of the reply-to queue. string   MQ queue name or queue URI
mq.retry.backoff.ms Wait time, in milliseconds, before retrying after retriable exceptions. long 60000 [0,…]
mq.message.mqmd.write Whether to enable a custom message builder to write MQ message descriptors. boolean false  
mq.message.mqmd.context Message context to set on the destination queue. This is required when setting some message descriptors. string   IDENTITY, ALL