Storage latency on a host fluctuates depending on hardware and workload conditions. By default, the cluster sets an alert trigger threshold far above the normal workload fluctuation value (2 seconds). When the alert is triggered, it typically indicates an anomaly in hardware or network environment associated with the host.
Host { .labels.name }: The average storage latency has exceeded { .threshold }.
The average storage latency of the physical disk pool {.id} on host { .labels.name } exceeds { .threshold }.
Host service performance degrades significantly.
Services inside virtual machines may frequently trigger high storage latency alerts.
Common causes are as follows. Check the physical disk latency and storage network latency monitoring data on AOC or the observability platform to determine the specific cause:
The NIC or optical module on the host is abnormal, causing an increase in packet loss rate and a sudden spike in network latency.
The HBA card or RAID card on the host is abnormal, causing a sharp increase in I/O latency across all physical disks, or even I/O unresponsiveness.
A physical disk on the host is abnormal, and the HBA card or RAID card has expanded the I/O impact scope while handling the disk anomaly.
This type of issue is typically accompanied by network anomalies, physical disk anomalies, or high latency alerts. Refer to the solution for the corresponding alert to address the issue. Typically, the host's storage latency alert clears once the issue is resolved.
