The alert is triggered when an internal service in the cluster fails to run normally.
Host { .labels.hostname }: The service process { .labels._service } is not running.
The service process { .labels._service } of host { .labels.hostname } is in an abnormal state: { .labels.state_str }.
SCVM { .labels.hostname }: The service process { .labels._service } is not running.
The service process { .labels._service } of SCVM { .labels.hostname } is in an abnormal state: { .labels.state_str }.
Witness node { .labels.hostname }: The service process { .labels._service } is not running.
The service process { .labels._service } of witness node { .labels.hostname } is in an abnormal state: { .labels.state_str }.
The impact of different service anomalies on the system varies. Refer to the alert description for the corresponding service for analysis.
Common causes include:
Maintenance operations
During a planned restart or upgrade, most services go through a stop-and-restart cycle, which may briefly trigger this alert. If the alert disappears on its own within a few minutes after the maintenance is complete, no action is needed.
Operating system anomaly
If the physical disk hosting the operating system fails, or the system becomes unresponsive, services may enter an abnormal state. If the system root partition enters a read-only or extremely slow state, the monitoring process on the node will also proactively shut down some services to ensure the stability of services operating in cluster mode.
Memory limit exceeded
During network anomalies, a large backlog of requests may cause a service process to exceed its allocated memory, triggering OOM detection by the system and proactively shutting down the service.
Hardware failure
Anomalies in hardware components in passthrough mode (such as RDMA NICs, encryption cards, or GPUs) may cause the related services to become abnormal.
In general, internal services in the system have built-in health checks and automatic restart capabilities, which allow them to restart and resume operation automatically after an anomaly.
If the alert appears during planned maintenance (such as a node restart or cluster upgrade), no action is typically needed. Wait for the operation to complete and observe for a few minutes; if the alert disappears on its own, no further action is required.
If the alert persists or recurs frequently, collect cluster logs during the anomaly period via AOC and submit them to an Arcfra after-sales engineer for further investigation.
