The alert is triggered when the storage service on a host is in an abnormal state.
Host { .labels.name }: The storage service health status is abnormal.
SCVM { .labels.name }: The storage service health status is abnormal.
The storage service health status of the physical disk pool {.id} of the host { .labels.name } is abnormal.
Virtual machines on the node can continue running; their I/O requests will be forwarded to other nodes in the cluster for processing, resulting in a slight performance decrease compared with the healthy state.
The overall storage capacity of the cluster decreases, typically in proportion to the number of abnormal nodes relative to the total number of nodes in the cluster.
Data on the node will become inaccessible. Data recovery will be triggered to rebuild the data on other healthy nodes, accompanied by data recovery alerts.
Storage service anomalies are mostly caused by missing hardware or physical disk failures. Common cases include:
NIC anomaly: When RDMA is enabled, the system injects RDMA NIC information into the storage service configuration file. If the NIC is found to be missing, the service cannot start and presents as a storage service anomaly.
Physical disk anomaly: The storage engine records physical disks mounted to the storage service in its own metadata. If a physical disk is removed, damaged, or cannot be read while the storage service is stopped (such as during a shutdown), the storage service will refuse to start because it cannot verify data integrity.
System disk failure: Anomalies on the physical disk hosting the operating system may cause storage services to be forcibly shut down due to the root partition becoming read-only or OS-level errors. If the disk remains missing or unreadable during restart, the system will also fail to start.
Check the storage service logs at /var/log/zbs/zbs-chunkd.* to confirm whether there are NIC or BDEV initialization error messages, and identify the associated storage device.
For recoverable or unintended hardware removal, such as a physical disk or NIC being accidentally disconnected, the storage service will automatically restart and clear the alert once the hardware is restored.
If the issue is caused by a NIC failure, contact an Arcfra after-sales engineer for NIC replacement, and then restart the service.
If the issue is caused by a physical disk failure, data on the affected node cannot be restored without loss, as its integrity cannot be self-verified. If the data on the node has already been recovered on other nodes, and there is no pending recovery data or inaccessible data anomalies in the cluster, contact an Arcfra after-sales engineer to reset the storage engine. Once the storage service returns to a healthy state, remount the remaining healthy physical disks on the node to resume normal operation.
