Short remaining physical disk lifetime

Description

Short remaining physical disk lifetime indicates that the disk is approaching its designed or expected end of life, and may be about to fail or experience performance degradation. This is an early signal that action should be taken to prevent data loss or service disruption.

Alert message

• Host { .labels.hostname }: The remaining lifetime of the physical disk { .labels._serial_number }({ .labels._device }) is less than { .threshold }.

• SCVM { .labels.hostname }: The remaining lifetime of the physical disk { .labels._serial_number }({ .labels._device }) is less than { .threshold }.

Impact

After the physical disk reaches the end of its lifespan, its risk of failure and possibility of performance degradation will increase significantly.

Cause

The direct cause is that the physical disk firmware reports insufficient remaining lifetime, which typically occurs when the cumulative total data written to the disk reaches 80% of the disk's designed threshold.

Troubleshooting approach

Run the smartctl -A command to check the remaining lifetime metrics for the physical disk.

• HDD and SATA or SAS SSD

  Run smartctl -A to obtain the disk lifetime metrics. An example is as follows:

  $smartctl -A /dev/sdc
  smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-1160.11.1.el7.arcfra.1.x86_64] (local build)
  Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
  
  === START OF READ SMART DATA SECTION ===
  SMART Attributes Data Structure revision number: 1
  Vendor Specific SMART Attributes with Thresholds:
  ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
    1 Raw_Read_Error_Rate     0x000f   130   130   039    Pre-fail  Always       -       4294967295
    5 Reallocated_Sector_Ct   0x0033   076   076   001    Pre-fail  Always       -       1
  ...
  177 Wear_Leveling_Count     0x0032   019   019   000    Old_age   Always       -       188982165529
  ...
  233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0

  The system uses the VALUE of attribute 233 as the remaining lifetime of the physical disk. If attribute 233 is not present, the VALUE of attribute 177 is used instead. An alert is triggered when the remaining lifetime falls below the threshold (default: 20%).

• NVMe SSD

  Run smartctl -A to obtain the NVMe SSD lifetime metrics. An example is as follows:

  $ smartctl -A /dev/nvme0n1
  smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.90-2307.3.0.oe1.arcfra.37.x86_64] (local build)
  Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
  
  === START OF SMART DATA SECTION ===
  SMART/Health Information (NVMe Log 0x02)
  Critical Warning:                   0x00
  Temperature:                        33 Celsius
  Available Spare:                    100%
  Available Spare Threshold:          10%
  Percentage Used:                    1%
  Data Units Read:                    1,000,847,365 [512 TB]
  Data Units Written:                 739,392,946 [378 TB]
  Host Read Commands:                 14,476,379,502
  Host Write Commands:                27,758,907,932
  Controller Busy Time:               15,756
  Power Cycles:                       88
  Power On Hours:                     14,660
  Unsafe Shutdowns:                   87
  Media and Data Integrity Errors:    0
  Error Information Log Entries:      1
  Warning  Comp. Temperature Time:    0
  Critical Comp. Temperature Time:    0
  Temperature Sensor 1:               46 Celsius

  The metrics related to disk remaining lifetime are Available Spare and Percentage Used. Since the two values differ in meaning, the remaining lifetime values they represent are not necessarily equal. Descriptions are as follows:

  • Available Spare: Spare capacity. A lower value indicates a shorter remaining lifetime of the NVMe SSD. This metric is used for the current alert. An alert is triggered when the value falls below the threshold (default: 20%).

  • Percentage Used: Wear level. A higher value indicates a shorter remaining lifetime of the NVMe SSD.

Solution

1. If the physical disk is confirmed to have insufficient remaining lifetime, unmount and replace the faulty disk.

2. After the replacement, manually resolve the alert.