MongoDB is a critical service on which system operations and virtualization scheduling depend. When the MongoDB service fails as a whole:
Virtualization scheduling tasks fail.
Maintenance operations cannot proceed normally.
The MongoDB service on the cluster is in an abnormal state.
When the MongoDB service fails as a whole, the cluster cannot provide services normally. All virtual machine lifecycle management operations are affected, including but not limited to virtual machine creation, power-on, snapshot, and hot migration. However, running virtual machines are not affected and virtual machine HA is not triggered.
MongoDB is a distributed cluster service. A total failure only occurs when the MongoDB services on more than half of the master nodes fail simultaneously. Common causes include:
Cluster startup in progress: Due to differences in startup time across nodes and services, MongoDB has not fully started yet. The alert will disappear on its own once all nodes have started; no action is required.
Storage network anomaly: The storage network between master nodes is unavailable or experiencing severe latency or jitter.
Network partition: Network isolation has occurred between nodes in the cluster.
Check connectivity between master nodes
Log in to a primary node and run ping -s 1400 <target IP> to test storage network connectivity between nodes.
If a system anomaly makes it impossible to determine the master nodes, run cat /etc/zbs/zbs.conf to view the IP address group in the mongo field (these are the nodes on which MongoDB should be running).
If significant packet loss is detected, investigate the network links one by one (NICs, switches, storage network, etc.).
Verify the MongoDB service status
After resolving network issues, check the service status of the MongoDB cluster instances. There should be one PRIMARY instance, with the remaining instances as SECONDARY.
If the status is abnormal, run systemctl restart mongod on the corresponding node to restart the service, and then check the status again.
