In an active-active cluster, hosts in either availability zone continuously probe the connectivity status with hosts in the other availability zone. When all hosts in one availability zone cannot connect to all hosts in the other availability zone, the situation is determined as an inter-zone network anomaly, and the alert is triggered.
{.labels.zone_name} is disconnected from { .labels.to_zone_name }'s storage network.
The P{ .quantile } worst round-trip latency within 5 minute(s) from the host { .labels.hostname } of { .labels.src_zone_name } to { .labels.target_zone_name } exceeds { .threshold }.
Increased network latency between availability zones directly impacts the cluster's I/O write performance and the response time for operations such as creating or editing virtual machines. When the round-trip latency exceeds 1 second or a connection interruption occurs:
Virtual machines with HA enabled and whose placement group rules allow them to run in the primary availability zone will trigger HA (if the primary availability zone has sufficient resources) and migrate to the primary availability zone.
Hosts and storage services in the secondary availability zone may become abnormal.
A large amount of data pending recovery appears in the cluster, and the recovery rate may remain at 0 for an extended period, causing data recovery to fail to complete.
High latency or interruption on the storage network between availability zones.
Since active-active clusters typically have a large number of nodes and each availability zone contains at least 2 hosts, a complete network disconnection between availability zones is usually not caused by a single NIC anomaly, but is more likely related to a failure at the network device or link level. Common possible causes include:
Uplink switch anomaly in both availability zones
Physical link interruption between the two availability zones (such as fiber failure)
Failure of switches or other relay devices used for inter-zone communication
Misconfigured network rules (such as ACL or firewall policies) that block communication
Address the issue based on the investigation results, for example:
Replace or repair the faulty switch, link, or relay device.
Adjust or restore incorrect network configurations.
Confirm and optimize redundancy of the inter-zone communication links to prevent single points of failure from occurring again.
