Event Gateway in Evicted state

Symptoms

When an Event Gateway instance is set up with ephemeral storage, it is possible for it to reach an Evicted state due to running out of space.

Causes

Kubernetes pods use ephemeral storage for scratch space, caching and for logs. The Event Gateway is spewing out logs that are not being cleaned up in time, which causes the Event Gateway to reach the Evicted state.

Resolving the problem

This issue can be resolved by adding more ephemeral storage for the Event Gateway, so that additional storage is available to clean up logs. You can add more storage by editing the EventGateway custom resource:

# excerpt from a Event Gateway CRD 
template:
  pod:
    spec:
      containers:
          name: egw
          resources:
            limits:
              ephemeral-storage: 500Mi

This updates the pod ephemeral storage limits for the Event Gateway, which prevents the Event Gateway instance from overloading the storage and being Evicted.