A physical disk in a "failing" state has not yet failed, but its performance has degraded, potentially posing a threat to system stability and service continuity.
In general, a physical disk is marked as failing when the system detects any of the following:
The I/O latency of the physical disk has increased but has not yet reached the timeout threshold.
The cumulative count of abnormal sectors on the physical disk exceeds the threshold (400).
Host { .labels.hostname }: The physical disk { .labels._serial_number }({ .labels._device }) is in sub-healthy state. Please remove the disk.
Host { .labels.hostname }: The physical disk { .labels._serial_number }({ .labels._device }) is in sub-health state. The system will automatically isolate the disk and restore the data to other healthy physical disks. Do not pull the disk.
SCVM { .labels.hostname }: The physical disk { .labels._serial_number }({ .labels._device }) is in sub-health state. The system will automatically isolate the disk and restore the data to other healthy physical disks. Do not pull the disk.
SCVM { .labels.hostname }: The physical disk { .labels._serial_number }({ .labels._device }) is in sub-healthy state. Please remove the disk.
The physical disk is at risk of damage or prolonged performance degradation, which may affect the performance of the underlying storage system and the smoothness of upper-layer services. When the system detects a failing disk, it initiates data migration and recovery for the data on that disk. It is recommended to unmount or replace the failing disk promptly.
The following causes trigger a failing disk alert. After the system completes data recovery or migration, the physical disk will no longer be in use. The alert will change to indicate that the disk has failed and is now in an unmounted state, and can be safely removed.
| Cause | Information to check |
|---|---|
| The disk-healthd service reports increased I/O latency on the disk, but it has not yet reached the timeout threshold. | On the node where the disk resides, view the disk-healthd logs: less /var/log/zbs/disk-healthd/disk-healthd.log, or run zbs-node show_disk_status /dev/sdx to view anomaly information for the disk. |
| S.M.A.R.T report: Reallocated Sector Count + Current Pending Sector Count ≥ 400 | Check the S.M.A.R.T information. |
Unmount and replace the faulty disk.
After the replacement, manually resolve the alert.
