KMS certificate expiration or connection anomaly

Description

When the certificate of the key management service (KMS) expires, or when the cluster is unable to establish a trusted connection with the KMS, encrypted volumes that rely on the external encryption service will not function normally.

Alert message

• The certificate for the key management service {.kms.name} will be expired in less than {.threshold} day(s).

• The certificate for the key management service {.kms.name} has expired.

• The cluster is unable to establish a trusted connection with the server {.ip} of the key management service {.kms.name}.

Impact

When the KMS certificate expires or the connection between the cluster and the KMS is abnormal, no new encrypted volumes can be created in the cluster. Existing encrypted volumes with active I/Os are typically not immediately affected in the short term. However, if the Meta Leader node in the cluster switches, all encrypted volumes may stop I/Os because they cannot obtain keys.

Cause

Possible causes include:

• The key management service certificate has expired.

• The connection between the key management service and the cluster is abnormal.

• The key management service itself undergoes an anomaly.

Solution

• If the certificate has expired, after obtaining a new certificate from the key management service, go to System configuration > Security > Key management service in AOC, select the corresponding key management service, and choose Upload certificate and private key to update the certificate.

• If the connection to the key management service is abnormal, first check whether the key management service itself is functioning normally:

  • If an anomaly is found, resolve it and observe whether the alert clears automatically.

  • If the key management service is normal but the alert persists, log in to the Meta Leader node in the cluster that uses the key management service and troubleshoot in the following order:

    • Check whether the network between the node and the key management service is functioning normally.

    • Obtain detailed error information about the key management service connection anomaly from the Meta Leader logs for further investigation.