The alert is triggered when the physical disk containing the operating system partition on a host fails, causing the file system to enter a read-only state to protect itself.
The system partition on the node { .labels.hostname } has entered the read-only state.
The storage service running on the node will be automatically terminated after a period of time and will no longer be able to provide services.
Other nodes may run abnormally because they cannot write logs or data.
Virtual machines in the cluster will trigger HA.
The physical disk containing the host's root partition has failed, experiencing I/O errors or an extended period of unresponsiveness.
Log in to the affected node.
Information:
If SSH remote login is unavailable because operating system logs cannot be written, try logging in via an out-of-band method such as IPMI.
Run the mount | grep " / " command (note: there is a space before and after the slash /). An example is as follows:
[root@zbs17-73 10:34:54 ~]$ mount |grep " / "
/dev/md127 on / type ext4 (ro,relatime)</td>If the output shows that the / directory is mounted in ro (read-only) mode, or if creating any file in the root partition returns a failure with a "Read-Only file system" message, the root partition has entered the read-only state.
Note:
When the root partition enters the read-only state, system logs may no longer be writable. Before proceeding, try to view
/var/log/messageand take a screenshot of the latest log content if possible. If the network is available, you can also copy the message file to another healthy node for future investigation.
When the root partition enters the read-only state, all services on the node are expected to run unstably. For virtual machines with HA configured, confirm that they have successfully failed over to other nodes via HA before restarting the node to restore services.
If the system fails to boot after the node restarts, the physical disk has experienced a serious failure. Log in to the node via IPMI to identify the faulty disk, and contact the hardware vendor for assistance.
If the system boots normally after the node restarts, the core OS data on the physical disk is intact. Use logs and physical disk error counts to identify the disk, and contact the hardware vendor for assistance.