Event Manager is in a pending state with EEM endpoints not ready for key exchange message

Symptoms

In Event Endpoint Management 11.3.0 and earlier, when an Event Manager instance is created with persistent storage, the Event Manager pods fail to start and the following message is displayed in the custom resource status:

status:
  conditions:
    - lastTransitionTime: '<timestamp>'
      message: EEM endpoints not ready for key exchange
      reason: Progressing
      status: 'True'
      type: Pending
    - lastTransitionTime: '<timestamp>'
      message: EEM endpoints not ready for key exchange
      reason: Progressing
      status: 'False'
      type: Ready

Causes

Event Manager storage detects the Unix lost+found folder in the directory that it is configured to write data to. Consequently, it does not include data encryption keys, and Event Manager waits for them to appear, causing the system to enter a pending state.

Resolving the problem

To resolve the problem, upgrade to Event Endpoint Management 11.3.1. If you are not ready to upgrade, complete the following steps to re-create the instance.

1. In the pod terminal, change the directory to where the lost+found file is located by using the following command: cd opt/storage.
2. Use ls to check that the lost+found file exists and that your data encryption keys were not created.
3. Remove the lost+found file by using the following command: rm -rf lost+found.
4. Restart the Event Manager pod to create a new instance.