Backing up and restoring

To back up and restore your Event Endpoint Management and Event Gateway instances, you can use a backup tool such as Velero. In addition, consider using a storage class that supports the Container Storage Interface (CSI) snapshotting (for example, the Ceph File System).

If you are running on the Red Hat OpenShift Container Platform, the OADP operator uses Velero and simplifies the installation of the backup software on your cluster, and the management of your backups and restorations.

On other Kubernetes platforms, Velero provides a Helm chart which you can use to install the software or a command line tool.

Follow these instructions to back up and restore your Event Endpoint Management and Event Gateway instances.

Before you begin

To back up and restore the Event Endpoint Management and Event Gateway instances, you must back up the following resources:

The main encryption key that is stored in a Kubernetes secret.
The data that is added to the instance. This is stored on Kubernetes Persistent Volumes (PVs) and within Kubernetes Persistent Volume Claims (PVCs).
Secrets containing certificates that are associated with the related instances.
The Event Endpoint Management and Event Gateway custom resource configurations.

This means that your backup location and configuration must be able to store both Kubernetes objects and volumes. For more information about the solutions you can use to back up PVs and PVCs, see the OpenShift Container Platform or Velero documentation. For example, you can use a remote object store such as AWS S3 and a CSI compliant storage class to create the PVC for your instance.

If you are on the OpenShift Container Platform and using the OADP operator to back up your instances, you must specify backup and snapshot configurations in the DataProtectionApplication custom resource. For more information about installing and configuring the DataProtectionApplication custom resource, see the OpenShift Container Platform documentation and select the storage type you want to configure.
In your CSI supported storage provider (such as Ceph), ensure you have the VolumeSnapshotClass configured for the CSI storage provider in your cluster.
To ensure that the correct items are backed up and restored properly, you can install the Velero client.

Backing up

If applicable, after your DataProtectionApplication is configured and other dependencies are prepared, you can back up your Event Endpoint Management and Event Gateway instances.

Create two Backup custom resources to back up your Event Endpoint Management and Event Gateway instances:

One for the main encryption key, secrets, and the PVCs containing the data of your instance deployments.
The other for the Event Endpoint Management and Event Gateway custom resource configurations.

Complete the following steps to create two Backup custom resources:

To ensure that only the main encryption key, secrets, and the PVCs that contain the data are configured for back up, and to easily separate them from the other resources that are related to the Event Endpoint Management instance, ensure that the events.ibm.com/backup: required label and the app.kubernetes.io/name: ibm-event-endpoint-management label are added to the following resources:
- The PVCs in the Event Endpoint Management instance namespace with names starting with manager-storage-<eem_instance_name>-ibm-eem-manager-.
- Any custom named PVC configured on the Event Endpoint Management instance.
- The secret in the Event Endpoint Management instance namespace with the name that ends with -mek-bak. For example, <eem_instance_name>-ibm-eem-mek-bak.
- Any custom secrets containing certificates added to the Event Endpoint Management instance as trustedCertificates.
In addition, add labels to the following secrets:
- If using the cert manager default configuration for the Event Endpoint Management and Event Gateway instances, add the events.ibm.com/backup: required and app.kubernetes.io/name: ibm-event-endpoint-management labels to secrets with the names <eem_instance_name>-ibm-eem-manager-ca, <eem_instance_name>-ibm-eem-manager, and <egw_instance_name>-ibm-egw-cert.
- If using custom secrets, add the events.ibm.com/backup: required and app.kubernetes.io/name: ibm-event-endpoint-management labels to all secrets supplied on the Event Endpoint Management and Event Gateway custom resource TLS configurations.
Note: If you want to include your user role mappings configuration in the backup, add the labels to the secret named <eem_instance_name>-ibm-eem-user-roles.
To ensure the Event Endpoint Management and Event Gateway custom resource configurations are configured for back up, add the events.ibm-event-endpoint-management.ibm.com/backup=required label to the following custom resources if not present:
- The Event Endpoint Management custom resource
- The Event Gateway custom resource

After the labels are applied, create and apply a Backup custom resource as follows that backs up secrets containing the encryption key and certificates, and the PVCs containing the data of your instance. Also, include the snapshots in the custom resource if you are using a CSI storage provider.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <unique_name>
  namespace: openshift-adp
spec:
  includedResources:
    - secrets
    - persistentvolumeclaims
    # To ensure the volume data is backed up when using CSI
    - volumesnapshots 
    - volumesnapshotcontents
  storageLocation: <backup_storage_location_cr_name>
  includeClusterResources: true
  labelSelector:
    matchLabels:
      app.kubernetes.io/name: ibm-event-endpoint-management
      events.ibm.com/backup: required
  includedNamespaces:
    - <event-endpoint-management-instance-namespace>
    - <event-gateway-instance-namespace>
  excludedResources: []

Note: Under spec.includedNamespaces, add all namespaces where Event Endpoint Management and Event Gateway instances are installed.

See the OADP and Velero documentation for more configuration options for the Backup custom resource.

After the Backup custom resource is applied, check whether the custom resource is updated with status information similar to the following snippet:

status:
  progress:
    itemsBackedUp: 11
    totalItems: 11
  csiVolumeSnapshotsCompleted: 1
  csiVolumeSnapshotsAttempted: 1
  phase: Completed

To ensure the resources specified in the Backup custom resource are backed up properly, run the following command:
```
velero backup describe <backup_name> --details -n openshift-adp
```

Create and apply another Backup custom resource as follows to back up the Event Endpoint Management and Event Gateway custom resource configurations.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <unique_name>
  namespace: openshift-adp
spec:
  defaultVolumesToRestic: false
  includeClusterResources: true
  storageLocation: <backup_storage_location_cr_name>
  includedNamespaces:
    - <event-endpoint-management-instance-namespace>
    - <event-gateway-instance-namespace>
  orLabelSelectors:
  - matchExpressions:
    - key: events.ibm-event-endpoint-management.ibm.com/backup
      operator: In
      values:
      - required

Note: Under spec.includedNamespaces, add all namespaces where Event Endpoint Management and Event Gateway instances are installed.

After the Backup of your Event Endpoint Management and Event Gateway custom resources is applied, check whether the Backup custom resource is updated with status information similar to the following:
```
status:
  progress:
    itemsBackedUp: 5
    totalItems: 5
  phase: Completed
```
To ensure your Event Endpoint Management and Event Gateway custom resource configurations are backed up properly, run the following command:
```
velero backup describe <backup_name> --details -n openshift-adp
```

If you are facing problems in creating a backup, see the troubleshooting information for OADP.

Important: The name of your Backup custom resource is required to restore your instance from the backup.

Restoring

Restoring your instance might be required for a number of reasons:

Loss of the entire Kubernetes cluster.
Loss of a namespace.
Loss of the Event Endpoint Management data or the encryption key.
To wind back the data to a previous state.
Loss of the Event Endpoint Management and Event Gateway instances.

You can restore the backed up Event Endpoint Management and Event Gateway instances as follows.

Before restoring your instance, the Event Endpoint Management operator must be installed again in the previously installed namespace if it was lost.

Note: In the namespace that you are restoring to, ensure that you do not have an instance with the same name as the one you are restoring. You cannot restore Event Endpoint Management and Event Gateway instances in the namespace if another instance with the same name already exists there. If you are trying to rewind your data, you must uninstall the Event Endpoint Management and Event Gateway instances you are restoring.

To restore your Event Endpoint Management and Event Gateway instances, complete the following steps:

To restore the encryption key secret, certificate secrets, and the PVC resources, create and apply a Restore custom resource similar to the following YAML:
```
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <unique_name>
  namespace: openshift-adp
spec:
  backupName: <name_of_the_backup_to_restore>
  existingResourcePolicy: update
  excludedResources: []
  includedResources: []
  restorePVs: true
```
Where <name_of_the_backup_to_restore> is the backup of the encryption key secret, certificate secrets, and the PVC resources.

See the OADP and Velero documentation for more configuration options for the Restore custom resource. For example, you can configure your custom resource to restore to an alternative namespace.
When you are applying this custom resource, the backup is loaded from the backup location and the <eem_instance_name>-ibm-eem-mek-bak secret, manager-storage-<eem_instance_name>-ibm-eem-manager-0 PVC and secrets containing TLS certificates are re-created.

Before proceeding to the next step, wait for the Restore custom resource to be applied and in Completed state, then check that the encryption key secret, the PVC resources and certficate secrets are created, and ensure that the resources are in Ready state.

Note: Velero removes some labels and annotations such as the volume.beta.kubernetes.io/storage-provisioner annotation when restoring PVCs. For some providers, this might leave the PVC in a Pending state. To fix this manually, add the volume.beta.kubernetes.io/storage-provisioner annotation back into the PVC with the same value as the volume.kubernetes.io/storage-provisioner annotation.

After the encryption key secret, TLS secrets, and the PVC resources are restored, and in Ready state, you can restore the Event Endpoint Management and Event Gateway custom resource configurations. To do this, create and apply another Restore custom resource similar to the following YAML:

 apiVersion: velero.io/v1
 kind: Restore
 metadata:
   name: <unique_name>
   namespace: openshift-adp
 spec:
   backupName: <name_of_the_backup_to_restore>
   includeClusterResources: true
   existingResourcePolicy: update
   hooks: {}
   includedNamespaces:
   - <event-endpoint-management-instance-namespace>
   - <event-gateway-instance-namespace>
   itemOperationTimeout: 1h0m0s
   orLabelSelectors:
   - matchExpressions:
     - key: events.ibm-event-endpoint-management.ibm.com/backup
       operator: In
       values:
       - required

Where <name_of_the_backup_to_restore>is the backup of the Event Endpoint Management and Event Gateway custom resource configurations.

Note: Under spec.includedNamespaces, add all namespaces where Event Endpoint Management and Event Gateway instances are installed.

After the Restore custom resource is applied and in Completed state, check that both the Event Endpoint Management and Event Gateway instances are created, and are in Running state.
Secure your access again by following guidance in the managing access and managing roles topics.

You can access the Event Endpoint Management and Event Gateway instances again with the same configuration and data as when the back up happened.