In an active-active cluster, the witness node continuously monitors the connectivity status between itself and the storage networks of the availability zones. When a network connection anomaly or interruption is detected, the alert is triggered.
The P{ .quantile } worst round-trip latency within 5 minute(s) from the witness node to { .labels.target_zone_name } exceeds { .threshold }.
The witness node is disconnected from { .labels.zone_name }'s storage network.
Witness node's state is unknown.
When all of the following conditions are met, a witness node network connection failure has no direct impact on cluster operation:
All nodes in both the primary and secondary availability zones are operating normally.
Network connectivity between the primary and secondary availability zones remains normal.
If the witness node remains in a network connection anomaly state and any of the following conditions also apply, the situation may lead to overall cluster failure:
An entire availability zone fails.
The network connection between the primary and secondary availability zones is abnormal.
The network status monitoring service triggers the alert when it detects abnormal network quality for the witness node or a complete disconnection of the witness node.
The witness node is typically deployed as a virtual machine at a location independent of both the primary and secondary availability zones. It is recommended to investigate in the following order:
Confirm whether the witness node is running.
Log in to the witness node and confirm whether it can connect normally to the storage networks of hosts in the availability zones.
On the witness node, probe the storage network addresses in the availability zones with large packets to check latency, jitter, and packet loss during large packet transmission. For example, run: ping -s 1400 1.1.1.1.
If the network is confirmed to be unreachable, contact the relevant network department based on the deployment location of the witness node to further confirm the network status.
The alert will be automatically cleared when the witness node resumes normal operation or the network connection between the witness node and the availability zones is restored.
