Insufficient cluster master nodes

Description

A master node refers to a node that runs the ZooKeeper and Meta services. ZooKeeper provides a distributed consensus protocol that enables coordination among multiple tasks in a distributed system. Meta provides metadata management services and is primarily responsible for managing storage objects, data replicas, access control, and data consistency.

The number of master nodes must meet the minimum redundancy requirements:

• When the cluster has no more than 4 nodes, 3 master nodes are expected.
• When the cluster has 5 or more nodes, 5 master nodes are expected.
• For an active-active cluster, it is typically expected that 2 master nodes are placed in each of the primary availability zone and the secondary availability zone.

If the number of master nodes falls below the expected level, the cluster's fault tolerance capability is affected, triggering the alert.

Alert message

There are { .labels.master_num } master nodes in the cluster, fewer than the expected { .labels.master_num_expected } master nodes.

Impact

Typically, the failure of a single master node does not affect cluster availability, while failures of multiple master nodes may affect cluster availability. If more than half of the cluster's master nodes are unavailable, the metadata service becomes unavailable and the cluster cannot function normally.

Using a 3-node and 5-node cluster as examples, the impact of failed master nodes in the cluster is shown in the following table:

Number of nodes	Number of failed master nodes	Impact
3	1	The cluster can provide services normally.
3	≥ 2	The cluster is unavailable.
5	≤ 2	The cluster can provide services normally.
5	≥ 3	The cluster is unavailable.

Cause

The number of configured master nodes in the cluster is lower than expected. Common causes include:

• Some nodes have failed or been manually removed.
• When a cluster with fewer than 5 nodes is expanded to 5 nodes for the first time, the number of master nodes may not have been adjusted yet.

Solution

• If the alert is caused by node removal:
  • If there are storage nodes in the cluster that meet the topology security requirements, you can convert them to master nodes until the expected number is reached.
  • If the cluster is in 2-node mode, add 1 master node as soon as possible; otherwise, the cluster will continue to operate in degraded mode.
• If the alert occurs during cluster scale-out, convert the storage nodes to master nodes after the scale-out is complete. The alert will clear automatically once the number of master nodes is restored to the expected level.