Symptoms
The Event Gateway fails to connect to the Event Endpoint Management Manager, and throws the following exception:
WARN com.ibm.ei.gateway.eem.core.BackendClient (EEM Backend Client) - [lambda$checkData$15:293] Error updating resources from https://<eem-manager-gateway-route>/clusters : Failed to create SSL connection
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
Causes
Certificates are automatically renewed by the Cert Manager if you are using it to generate TLS certificates. The Event Endpoint Management Manager and the Event Gateway pick up the renewed CA certificate. If the Event Endpoint Management Manager or the Event Gateway renews at a different time, the Event Endpoint Management Manager and the Event Gateway get a different CA certificate.
To confirm this issue, examine the ca.crt in the Event Endpoint Management Manager secret <instance_name>-ibm-eem-manager, and compare to the secret in the Event Gateway<instance_name>-ibm-egw-cert.
Resolving the problem
Delete both the Event Endpoint Management Manager and the Event Gateway secrets to allow the certificates to be renewed by the Cert Manager.
You might see the following error around 5 minutes after you deleted both the Event Endpoint Management Manager and the Event Gateway secrets:
WARN com.ibm.ei.gateway.eem.core.BackendClient (EEM Backend Client) - [lambda$checkData$15:293] Error updating resources from https://quick-start-manager-ibm-eem-gateway-eim.apps.tag.cp.fyre.ibm.com/clusters : javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
If you see this error, delete the Event Endpoint Management Manager pod. After the Event Endpoint Management Manager becomes Ready again, the Event Gateway will reconnect to the Event Endpoint Management Manager.