Physical disk S.M.A.R.T. check failure

Description

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) is an automated status monitoring and early warning system and specification. It monitors and records the condition of physical disk hardware, including platters and circuitry, using built-in diagnostic commands, and compares the results against preset safety thresholds defined by the manufacturer. If the monitored values approach or exceed these thresholds, this alert is triggered. When a physical disk fails a S.M.A.R.T. check, it may still function but is no longer reliable and could fail at any time. Unless the anomaly is confirmed to be a temporary condition caused by external factors, such as a transient high temperature in the data center, the disk should be replaced.

Alert message

• Host { .labels.hostname }: Physical disk { .labels._serial_number }({ .labels._device }) S.M.A.R.T. check failed.

• SCVM { .labels.hostname }: Physical disk { .labels._serial_number }({ .labels._device }) S.M.A.R.T. check failed.

Impact

When a physical disk fails the S.M.A.R.T. check, it typically indicates potential performance issues with the disk, and the risk of subsequent failure will increase significantly.

Cause

The physical disk firmware reports that the disk is in an abnormal state. The specific cause needs to be confirmed by providing the S.M.A.R.T. report to the disk vendor.

Information:

A high temperature inside the server chassis can cause the physical disk S.M.A.R.T. check to fail. Once the temperature returns to normal, the output of smartctl -H will return to normal. In this case, no disk replacement is required. However, the alert will remain triggered. After logging in to the physical node where the disk resides, you can run the zbs-node set_disk_healthy command to reset the disk fault record. The alert will be automatically resolved after a short period of time.

Troubleshooting approach

You can use the host_disk_temperature_celsius or scvm_disk_temperature_celsius physical disk temperature metrics in the monitoring system to determine whether the S.M.A.R.T. check failure is caused by high temperature.

Run the smartctl -a command to obtain the S.M.A.R.T. check results for the physical disk. Under normal conditions, the self-test result is passed. When an anomaly occurs, it may display a corresponding fault message or FAILED.

Solution

1. If the physical disk S.M.A.R.T. check is confirmed to have failed, unmount and replace the faulty disk.

2. After the replacement, manually resolve the alert.