The system periodically uses fping to check network connectivity and latency between hosts. When the latency exceeds the configured threshold, the alert is triggered.
The storage network 5-minute P{ .quantile } ping response time is greater than { .threshold } between the two hosts: { .labels.hostname } and { .labels.to_hostname }.
The storage network 5-minute P{ .quantile } ping response time is greater than { .threshold } between the two SCVMs: { .labels.hostname } and { .labels.to_hostname }.
When latency only slightly exceeds 5 ms (such as tens of milliseconds), overall cluster performance is slightly degraded; the extent depends on the magnitude and duration of the latency.
When latency rises to hundreds of milliseconds, the cluster may experience occasional data recovery.
When latency reaches over several hundred milliseconds, communication between cluster nodes is severely impaired, which may cause the cluster to stop working normally.
Network latency is usually caused by network jitter or traffic burst. If high latency persists for a long time or jitter occurs frequently, a comprehensive analysis considering the following factors is needed, for example:
Excessive overall system load or anomalies in virtual network components.
Failure or improper configuration of physical network devices (switch, cable, optical module, etc.).
Use latency measurement tools (such as fping and ping) to check latency fluctuations.
Capture packets on physical network ports to analyze the specific cause.
Check whether the physical network links (switch, NIC, cable, etc.) are abnormal.
Investigate the configuration and load of virtualized network components such as OVS and NIC drivers.
Based on the measurement and analysis results, optimize the network architecture or replace hardware.
