Event Manager fails with Unable to find main key backup error

Symptoms

When an Event Manager instance is created, Event Manager pods fail to reconcile with the following error in the operator logs:

Caused by: com.ibm.eem.dependents.EEMSecretMainEncryptionKey$NoMainKeyBackupException: Unable to find main key backup
	at com.ibm.eem.dependents.EEEMSecretMainEncryptionKey.retrieveKeyBackup(EPSecretMainEncryptionKey.java:129)
	at com.ibm.eem.EventEndpointManagementReconciler.reconcile(EventEndpointManagementReconciler.java:223)

The following status is displayed in the Event Manager custom resource:

status:
  conditions:
  - lastTransitionTime: '2024-02-26T19:49:43Z'
    message: Unable to find main key backup
    reason: UnhandledException
    status: 'True'
    type: Error

Causes

When the Event Manager pod is started, a handshake takes place between the pod and the operator to initialize storage encryption. If the Event Manager pod restarts while initializing the encryption, the handshake might not complete successfully, leaving the Event Manager instance and the operator out-of-sync. As a result, the Event Manager pod never goes into a Ready state.

Resolving the problem

To resolve the problem, recreate the instance as follows:

1. Delete the existing Event Manager instance.
2. Delete the PersistentVolumeClaim associated with the instance.
3. Create an Event Manager instance.