Migrating existing applications to the Event Streams schema registry

If you are using the Confluent Platform schema registry, Event Streams provides a migration path for moving your Kafka consumers and producers over to use the Event Streams schema registry.

Migrating schemas to Event Streams schema registry

To migrate schemas, you can use schema auto-registration in your Kafka producer, or you can manually migrate schemas by downloading the schema definitions from the Confluent Platform schema registry and adding them to the Event Streams schema registry.

Migrating schemas with auto-registration

When using auto-registration, the schema will be automatically uploaded to the Event Streams schema registry, and named with the subject ID (which is based on the subject name strategy in use) and a random suffix.

Auto-registration is enabled by default in the Confluent Platform schema registry client library. To disable it, set the auto.register.schemas property to false.

Note: To auto-register schemas in the Event Streams schema registry, you need an API key that has operator role permissions (or higher) and permission to create schemas. You can generate API keys by using the ES UI or CLI.

Using the UI:

Log in to your Event Streams UI as an administrator.
Click Connect to this cluster on the right.
Enter a name into the API key generation tool and click Produce, consume, create topics and schemas.
Enter your topic name or set All topics to on.
Enter the consumer group name and click Generate to generate an API key.

Using the CLI:

Log in to your cluster as an administrator by using the IBM Cloud Private CLI:
cloudctl login -a https://<Cluster Master Host>:<Cluster Master API Port>
The master host and port for your cluster are set during the installation of IBM Cloud Private.
Run the following command to initialize the Event Streams CLI on the cluster:
cloudctl es init
Run the following command to create a service ID with an API key that has permissions to auto-register schemas and produce messages to any topic:
cloudctl es iam-service-id-create <service-id-name> --role operator --all-topics --all-schemas

Migrating schemas manually

To manually migrate the schemas, download the schema definitions from the Confluent Platform schema registry, and add them to the Event Streams Schema Registry. When manually adding schemas to the Event Streams Schema Registry, the provided schema name must match the subject ID used by the Confluent Platform schema registry subject name strategy.

If you are using the default TopicNameStrategy, the schema name must be <TOPIC_NAME>-<'value'|'key'>

If you are using the RecordNameStrategy, the schema name must be <SCHEMA_DEFINITION_NAMESPACE>.<SCHEMA_DEFINITION_NAME>

For example, if you are using the default TopicNameStrategy as your subject name strategy, and you are serializing your data into the message value and producing to the MyTopic topic, then the schema name you must provide when adding the schema in the UI must be MyTopic-value

For example, if you are using the RecordNameStrategy as your subject name strategy, and the schema definition file begins with the following, then the schema name you must provide when adding the schema in the UI must be org.example.Book:

{
    "type": "record",
    "name": "Book",
    "namespace": "org.example",
    "fields": [
...

If you are using the CLI, run the following command when adding the schema:

cloudctl es schema-add --create --name org.example.Book --version 1.0.0 --file /path/to/Book.avsc

Migrating a Kafka producer application

To migrate a Kafka producer application that uses the Confluent Platform schema registry, secure the connection from your application to Event Streams, and add additional properties to enable the Confluent Platform schema registry client library to interact with the Event Streams schema registry.

Configure your producer application to secure the connection between the producer and Event Streams.
Retrieve the full URL for the Event Streams API endpoint, including the host name and port number by using the cloudctl es init command.

Ensure you add the following schema properties to your Kafka producers:

Property name	Property value
`schema.registry.url`	`https://<host name>:<API port>`
`basic.auth.credentials.source`	`SASL_INHERIT`

You can also use the following code snippet for Java applications:

props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "https://<host name>:<API port>");
props.put(AbstractKafkaAvroSerDeConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "SASL_INHERIT");

Set the Java SSL truststore JVM properties to allow the Confluent Platform schema registry client library to make HTTPS calls to the Event Streams schema registry. For example:
```
export KAFKA_OPTS="-Djavax.net.ssl.trustStore=/path/to/es-cert.jks \ 
      -Djavax.net.ssl.trustStorePassword=password"
```

Migrating a Kafka consumer application

To migrate a Kafka consumer application that uses the Confluent Platform schema registry, secure the connection from your application to Event Streams, and add additional properties to enable the Confluent Platform schema registry client library to interact with the Event Streams schema registry.

Configure your consumer application to secure the connection between the consumer and Event Streams.
Retrieve the full URL for the Event Streams API endpoint, including the host name and port number by using the cloudctl es init command.

Ensure you add the following schema properties to your Kafka producers:

Property name	Property value
`schema.registry.url`	`https://<host name>:<API port>`
`basic.auth.credentials.source`	`SASL_INHERIT`

You can also use the following code snippet for Java applications:

props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "https://<host name>:<API port>");
props.put(AbstractKafkaAvroSerDeConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "SASL_INHERIT");

Set the Java SSL truststore JVM properties to allow the Confluent Platform schema registry client library to make HTTPS calls to the Event Streams schema registry. For example:
```
export KAFKA_OPTS="-Djavax.net.ssl.trustStore=/path/to/es-cert.jks \ 
       -Djavax.net.ssl.trustStorePassword=password"
```