When services related to cluster monitoring and alerting are running abnormally, the corresponding alerts are triggered. These anomalies may cause metric collection, alert sending, the Prometheus service, or the Siren service to fail to work normally, thereby affecting the monitoring and alerting features on AOC.
Cluster scraping-metric service does not work.
The alert-sending function of the node { .labels.hostname } is in an abnormal state.
The Prometheus service on the cluster is in an abnormal state.
The Siren service on the node { .labels.hostname } is in an abnormal state.
Metric collection service anomaly
This will cause the entire monitoring and alerting system to fail. The monitoring views in AOC will fail to load monitoring data and alerts.
Node alert-sending anomaly
AOC cannot display or forward alert information as expected.
Prometheus service anomaly
Local cluster monitoring data collection and alert sending are affected, meaning clusters not associated with an observability service on AOC cannot display monitoring data or trigger alerts.
Siren service anomaly
Cluster alerts cannot be displayed or sent externally; configured alert notification channels cannot receive alert notifications normally; clusters not associated with an observability service cannot trigger alerts; clusters associated with an observability service cannot receive alerts forwarded from local cluster monitoring, etc.
The cause for each alert is as follows:
Metric collection service anomaly
The vmagent service on the cluster's Octopus Leader node is not running or keeps restarting.
Node alert-sending anomaly
The siren service on the master node (Octopus Leader or the node hosting advanced monitoring) has not received a Prometheus heartbeat for 2 minutes.
Prometheus service anomaly
The Siren service on the node checks the Prometheus service in the primary node or the advanced monitoring virtual machine every 1 minute. If it is unable to respond to requests for 5 consecutive minutes, the service may already be in an abnormal state, triggering this alert.
Siren service anomaly
The Siren service on the cluster's Siren Leader node is in an abnormal state, or other nodes have detected an anomaly in the Siren Leader's Siren service through the management or storage network.
Metric collection service anomaly
Run zbs-cluster show_election_servers -s octopus to locate the Octopus Leader node.
On the Octopus Leader node, run systemctl status vmagent to check the service status. If the service is not running or keeps restarting, try running systemctl restart octopus to restart vmagent.
If the service cannot recover, collect cluster logs via AOC and contact an Arcfra after-sales engineer.
Node alert-sending anomaly
Log in to the current Octopus Leader node or the node where advanced monitoring resides.
Run zbs-cluster show_election_servers -s octopus to locate the Octopus Leader node.
Check whether the node's siren logs show that alerting_is_normal has not been received for 2 consecutive minutes.
If advanced monitoring is deployed: check whether the virtual machine file system is read-only; try restarting the virtual machine to fix the issue; then check the Prometheus logs in the virtual machine container to rule out network jitter.
If advanced monitoring is not deployed: check whether a "Prometheus service anomaly" alert also exists; if any, resolve that alert first.
If the issue persists, collect cluster logs via AOC and contact an Arcfra after-sales engineer.
Prometheus service anomaly
Check whether the Prometheus service is running normally on the Meta Leader node, the Octopus Leader node, and the advanced monitoring virtual machine.
In the Siren logs, filter for Prometheus service anomaly-related entries using the keyword prometheus_is_normal, then take targeted action based on the error cause.
Siren service anomaly
On the host where the anomaly is reported, run systemctl status siren to check the Siren service status.
Try running systemctl restart siren to restart the Siren service and observe whether it recovers.
If the issue persists, collect cluster logs via AOC and contact an Arcfra after-sales engineer.
