SEL hardware alert mechanism

Hardware high availability plays a crucial role in modern data centers, cloud platforms, and critical business systems, since a single hardware failure can cause significant financial loss. However, many hardware issues develop no obvious symptoms in their early stages, making it hard to detect potential failures promptly through traditional manual inspections.

System Event Log (SEL) and Sensor Data Record (SDR) data from the Baseboard Management Controller (BMC) lack a unified analysis platform, and fault localization largely relies on the experience of operations personnel. Moreover, traditional operations react to failures only after they occur, hindering predictive maintenance.

With SEL event (such as temperature exceeding thresholds or insufficient power redundancy) and SDR sensor metadata (such as thresholds and readings) collection, analysis, and alert generation, you can achieve full lifecycle management of hardware health, timely identify hardware issues, and proactively replace faulty components.

Architecture and components

• Hardware server

  The hardware server is deployed in AOC as a Pod. It periodically queries the clusters associated with AOC in round-robin mode to collect the information of agents, and records their deployment status.

• Agent

  Agents are deployed on each node of the cluster, and are mainly divided into the following types:

  • ipmi-agent: Collects SEL events and SDR sensor data through in-band mode, parses the data, and stores it in a local SQLite database. It determines whether SEL events should trigger alerts and provides query interfaces for SEL and SDR.

  • alert-proxy: Receives alerts sent by ipmi-agent, queries the alert rules interface in round-robin mode, determines whether to enable the alert rules, and decides whether to forward alerts to the cluster alerting components.

Data processing workflows

It takes about 1 to 6 minutes from the generation of an SEL event until you receive its alert.

Collection process

SEL

1. ipmi-agent runs the ipmitool -vv sel list command to obtain the raw SEL data in hexadecimal format;

2. ipmi-agent uses a third-party library to parse the raw SEL hexadecimal data into an SEL event;

3. ipmi-agent stores the SEL event in the local SQLite database, keeping the latest 3,000 events by timestamp.

SDR

1. ipmi-agent runs the ipmitool -vv sdr command to obtain the raw SDR data in text format;

2. ipmi-agent parses the raw SDR text data line by line into SDR data;

3. ipmi-agent stores the SDR data in the local SQLite database, without any limit on the number of entries.

Alert process

1. After storing the SEL event in the SQLite database, ipmi-agent checks whether the event matches the alert rules. If matched, it sends an alert to alert-proxy;

2. alert-proxy periodically queries the alert rule interface in round-robin mode and determines whether to enable the alert rules. If alert rules are to be enabled, it forwards the alert to the cluster alert component.

Query process

1. ipmi-agent provides SEL and SDR query interfaces. AOC queries SEL and SDR data through these interfaces and displays the results.

2. ipmi-agent also supports request proxy, forwarding query requests to other nodes.

Data display

SEL fields

SDR fields

Alert rules

Note:

• To avoid alert storms, instead of triggering new alerts, the following situations only update existing alerts.
  • The same sensor reports different types of power sensor status abnormalities.
  • The same sensor reports different types of power redundancy status abnormalities.
  • The same sensor reports different types of disk slot drive failures.
  • The same sensor reports different types of memory redundancy status abnormalities.
  • The same sensor reports different types of memory status abnormalities.
  • The same sensor reports different types of power module redundancy status abnormalities.
• The same sensor refers to the sensor with the same Sensor Number and Sensor Type Code.

Raw SEL fields

Raw SDR fields

The following table takes the full sensor record as an example. Other types of SDR record formats are similar.

ipmitool -vv sdr output example

Sensor ID              : Inlet Temp (0x5)
 Entity ID             : 64.96 (Air Inlet)
 Sensor Type (Threshold)  : Temperature (0x01)
 Sensor Reading        : 29 (+/- 1) degrees C
 Status                : ok
 Nominal Reading       : 23.000
 Normal Minimum        : 11.000
 Normal Maximum        : 69.000
 Upper critical        : 47.000
 Upper non-critical    : 42.000
 Lower critical        : -7.000
 Lower non-critical    : 3.000
 Positive Hysteresis   : 2.000
 Negative Hysteresis   : 2.000
 Minimum sensor range  : Unspecified
 Maximum sensor range  : Unspecified
 Event Message Control : Per-threshold
 Readable Thresholds   : lcr lnc unc ucr 
 Settable Thresholds   : lnc unc 
 Threshold Read Mask   : lcr lnc unc ucr 
 Assertion Events      : 
 Assertions Enabled    : lnc- lcr- unc+ ucr+ 
 Deassertions Enabled  : lnc- lcr- unc+ ucr+