Symptoms
When an Event Gateway instance is set up with ephemeral storage, it is possible for it to reach an Evicted state due to running out of space.
Causes
Kubernetes pods use ephemeral storage for scratch space, caching and for logs. The Event Gateway is spewing out logs that are not being cleaned up in time, which causes the Event Gateway to reach the Evicted state.
Resolving the problem
This issue can be resolved by giving the Event Gateway more ephemeral storage so that it is given a bit more leeway to clean up logs. This can be done by editing the Event Gateway custom resource:
# excerpt from a Event Gateway CRD
template:
pod:
spec:
containers:
name: egw
resources:
limits:
ephemeral-storage: 500Mi
This updates the pod ephemeral storage limits for the Event Gateway, which prevents the Event Gateway instance from overloading the storage and being Evicted.