Migrating the Kafka Bridge and REST producer to Confluent REST Proxy

The Confluent REST Proxy replaces both the Event Streams Kafka Bridge and the REST producer. To complete this migration, deploy the Confluent REST Proxy, recreate your client credentials, and update your client applications to use the new endpoint and API. There is no data to migrate.

To migrate from the Event Streams Kafka Bridge or REST producer to the Confluent REST Proxy, complete the following steps:

1. Deploy the REST Proxy and update credentials
2. Migrate Kafka Bridge applications
3. Migrate REST producer applications
4. Verify your setup

Background

Event Streams provides two HTTP-based interfaces to connect to Kafka:

• Kafka Bridge: An interface that translates HTTP requests to Kafka operations, supporting producing and consuming messages, topic and partition metadata, and consumer group management.
• REST producer: A lightweight, produce-only interface for applications that only need to publish messages.

When migrating to Confluent for Kubernetes, the Confluent REST Proxy replaces both the Kafka Bridge and the REST producer with an equivalent HTTP interface to Kafka, which means that existing REST-based integrations can continue with minimal changes.

Note: The Event Streams REST producer only produces messages. The Confluent REST Proxy also supports consuming messages and administrative operations, so there is no loss of capability.

Deploying the REST Proxy and updating credentials

Before migrating your applications, deploy the Confluent REST Proxy and ensure that your clients can authenticate and reach the new endpoint:

1. Deploy the REST Proxy by creating a KafkaRestProxy custom resource configured with your selected authentication mechanism over TLS. For configuration details, see the Confluent REST Proxy documentation.

2. Configure client authentication on the REST Proxy. Switch to a supported authentication mechanism such as HTTP Basic authentication, OAuth/OIDC, or mTLS. Create a Kubernetes secret containing the credentials and reference it in the KafkaRestProxy custom resource. For the full list of supported mechanisms, see the Confluent REST Proxy security documentation.

3. Update the base URL, protocol, and credentials in every client application. Every client must update the following, regardless of whether it uses the Kafka Bridge or the REST producer:

   What changes	Currently in Event Streams	Replace with
   Host and port	http://<bridge-or-rest-producer-svc>:<port>	http(s)://<kafka-rest-proxy-svc>:<port>
   Protocol	HTTP	HTTP or HTTPS, depending on your KafkaRestProxy deployment
   Credentials (Kafka Bridge clients)	Kafka Bridge does not authenticate client requests.	Configure credentials matching the authentication mechanism on the REST Proxy. For more information, see deploying the REST Proxy and updating credentials.
   Credentials (REST producer clients)	SCRAM-SHA-512 (HTTP Basic) or mTLS	Credentials matching the authentication mechanism configured on the REST Proxy. For more information, see deploying the REST Proxy and updating credentials.

   Important: Confirm the protocol (HTTP or HTTPS), hostname, and port before updating your clients, as the values depend on how the KafkaRestProxy custom resource was deployed.

Migrating Kafka Bridge applications

If your application uses the Event Streams Kafka Bridge, most v2 API endpoints carry over unchanged. Review the following compatibility table to identify which endpoints require changes, then make the updates described in the following sections.

API compatibility overview

The following table summarizes the compatibility of each Kafka Bridge endpoint with the Confluent REST Proxy v2 API:

Endpoints marked as Additive have extra fields in the Confluent response that are not present in the Kafka Bridge response. If your client validates the response structure strictly, update it to allow additional fields.

For the remaining endpoints, make the updates described in the following sections:

Updating health check endpoints

The Confluent REST Proxy does not expose the Kafka Bridge health check paths. Update the health check endpoints in your application or deployment configuration as follows:

A 200 response from GET /v3/clusters confirms that the proxy is reachable and connected to Kafka. Check the status code only. Do not rely on the response body content.

Handling the missing partitions field in subscription responses

The Kafka Bridge returns both the subscribed topics and the assigned partitions in the GET /subscription response. The Confluent REST Proxy returns only the subscribed topic names. If your application reads response.partitions, update it to handle a missing field. The value will be undefined or null.

For example, the following response is returned by the Kafka Bridge:

GET /consumers/my-group/instances/my-instance/subscription
→ HTTP 200
{
  "topics": [
    "orders"
  ],
  "partitions": [
    {
      "orders": [
        0,
        1,
        2
      ]
    }
  ]
}

The Confluent REST Proxy returns the following response for the same request:

GET /consumers/my-group/instances/my-instance/subscription
→ HTTP 200
{
  "topics": [
    "orders"
  ]
}

Handling the missing timestamp field in poll responses

The Kafka Bridge appends a timestamp (in epoch milliseconds) to every record in the poll response. The Confluent REST Proxy v2 does not include this field. If your application reads record.timestamp, update it to handle a missing field. The value will be undefined or null.

For example, the following response is returned by the Kafka Bridge:

GET /consumers/my-group/instances/my-instance/records
→ HTTP 200
[{"topic": "orders", "key": null, "value": {"orderId": 1001},
  "partition": 0, "offset": 42, "timestamp": 1700000000000}]

The Confluent REST Proxy returns the following response for the same request:

GET /consumers/my-group/instances/my-instance/records
→ HTTP 200
[{"topic": "orders", "key": null, "value": {"orderId": 1001},
  "partition": 0, "offset": 42}]

Handling the changed HTTP status code for offset commits

The Kafka Bridge returns HTTP 204 No Content with an empty body on a successful commit. The Confluent REST Proxy returns HTTP 200 OK with an empty JSON array body []. If your client checks for exactly 204, update it to accept 200 as a successful response.

For example, the Kafka Bridge returns the following response on a successful commit:

POST /consumers/my-group/instances/my-instance/offsets
→ HTTP 204 No Content  (empty body)

The Confluent REST Proxy returns the following response for the same request:

POST /consumers/my-group/instances/my-instance/offsets
→ HTTP 200 OK
[]

Important: The [] body is not formally specified and might change in a future release. Check the HTTP status code only. Any HTTP 200 response indicates a successful commit.

Migrating REST producer applications

The Event Streams REST producer is a produce-only interface that uses POST /topics/{topic}/records. Unlike Kafka Bridge applications, REST producer applications must update the request path, body structure, and Content-Type header.

Review the following table for a full summary of changes, then apply the updates to your application:

Required API changes

For example, the following request is sent by the Event Streams REST producer:

POST /topics/orders/records
Content-Type: application/json

{"orderId": 1001, "item": "widget"}

The equivalent request for the Confluent REST Proxy is:

POST /topics/orders
Content-Type: application/vnd.kafka.json.v2+json

{"records": [{"value": {"orderId": 1001, "item": "widget"}}]}

You must update the path, body structure, and Content-Type header. SCRAM-SHA-512 is not supported by the Confluent REST Proxy. Recreate and configure your credentials on the REST Proxy by using a supported authentication mechanism as described in deploying the REST Proxy and updating credentials.

If you are producing multiple records in a single request or by using schemas, review the following sections:

Producing multiple records in a single request

The Confluent REST Proxy accepts multiple records in a single request. To produce multiple records, include them all in the records array. For example:

POST /topics/orders
Content-Type: application/vnd.kafka.json.v2+json

{
  "records": [
    {"key": "key-1", "value": {"orderId": 1001}},
    {"key": "key-2", "value": {"orderId": 1002}}
  ]
}

Producing messages with a schema

If your Event Streams REST producer used the schemaname and schemaversion query parameters, you must switch to referencing schemas by their Schema Registry ID. The schemaname, schemaversion, and schemavalidation query parameters are not supported by the Confluent REST Proxy.

Confluent REST Proxy integrates with Schema Registry and supports three schema formats: Avro (avro), JSON Schema (jsonschema), and Protobuf (protobuf). Specify the schema format in the Content-Type header to determine how the REST Proxy serializes the message before writing it to Kafka.

The REST Proxy must be configured with schema.registry.url to communicate with Schema Registry. Schema Registry validates and stores the schema. The REST Proxy uses it to serialize records before producing to Kafka. For more information, see the Confluent Schema Registry documentation.

Use one of the following options to reference a schema in your produce request:

• Option 1: Register the schema in Schema Registry first, then reference it by its numeric ID in your produce requests.
• Option 2: Include the schema directly in the request body. Schema Registry registers it automatically and returns an ID that you can reuse in subsequent requests.

Option 1: Produce by using a pre-registered schema ID

Register the schema in Schema Registry first and use the returned numeric ID to reference it in your produce requests. The REST Proxy validates each record against the schema before writing to Kafka.

Complete the following steps:

1. Register the schema in Schema Registry and note the numeric ID that is returned. For example:

   POST /subjects/orders-value/versions
   Content-Type: application/vnd.schemaregistry.v1+json
   
   {
     "schema": "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"orderId\",\"type\":\"int\"},{\"name\":\"item\",\"type\":\"string\"}]}"
   }
   

   Note the id field in the response. For example:

   {"id": 3}
   

2. Specify that ID in value_schema_id in your produce request. For example:

   POST /topics/orders
   Content-Type: application/vnd.kafka.avro.v2+json
   
   {
     "value_schema_id": 3,
     "records": [{"value": {"orderId": 1001, "item": "widget"}}]
   }
   

   The response echoes value_schema_id and the offsets field confirms where the record was written. For example:

   {
     "offsets": [{"partition": 0, "offset": 42, "error_code": null, "error": null}],
     "key_schema_id": null,
     "value_schema_id": 3
   }
   

   If a record does not match the schema, the REST Proxy rejects the request with HTTP 400 before any message reaches Kafka. For example:

   {"error_code": 400, "message": "Bad Request: Field \"orderId\" content mismatch: Expected int. Got VALUE_STRING"}
   

Option 2: Produce by using an inline schema

Include the full schema as a JSON string in the value_schema field of your request body. Schema Registry registers it automatically and returns the assigned value_schema_id in the response. Use that ID in subsequent requests instead of resending the full schema. For example:

POST /topics/orders
Content-Type: application/vnd.kafka.avro.v2+json

{
  "value_schema": "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"orderId\",\"type\":\"int\"},{\"name\":\"item\",\"type\":\"string\"}]}",
  "records": [{"value": {"orderId": 1001, "item": "widget"}}]
}

The response includes the assigned value_schema_id. For example:

{
  "offsets": [{"partition": 0, "offset": 43, "error_code": null, "error": null}],
  "key_schema_id": null,
  "value_schema_id": 4
}

If the message does not match the schema, the REST Proxy rejects the request with HTTP 400.

Verifying your setup

After completing the previous steps, verify that the REST Proxy is working correctly by testing metadata retrieval and message production through the new endpoint. Use the endpoints listed in the API compatibility overview.