Event Processing fails with Unable to find main key backup error

Symptoms

When an Event Processing instance is created, Event Processing pods fail to reconcile with the following error in the operator logs:

com.ibm.ei.dependents.EPSecretMainEncryptionKey$NoMainKeyBackupException: Unable to find main key backup
	at com.ibm.ei.dependents.EPSecretMainEncryptionKey.retrieveKeyBackup(EPSecretMainEncryptionKey.java:132)
	at com.ibm.ei.EventProcessingReconciler.reconcile(EventProcessingReconciler.java:226)

The following status is displayed in the Event Processing custom resource:

status:
  conditions:
  - lastTransitionTime: '2024-02-26T19:49:43Z'
    message: Unable to find main key backup
    reason: UnhandledException
    status: 'True'
    type: Error

Causes

When the Event Processing pod is started, a handshake takes place between the pod and the operator to initialize storage encryption. If the Event Processing pod restarts while initializing the encryption, the handshake might not complete successfully, leaving the Event Processing instance and the operator out-of-sync. As a result, the Event Processing pod never goes into a Ready state.

Resolving the problem

To resolve the problem, recreate the instance as follows:

1. Delete the existing Event Processing instance.
2. Delete the PersistentVolumeClaim associated with the instance.
3. Create an Event Processing instance.