ACOS clusters provide a monitoring and analysis feature that monitors metrics for various resource objects in the cluster and organizes and presents data in views and charts. Operations personnel can use monitoring and analysis to understand the usage of key resources such as CPU, memory, networking, and storage. In failure scenarios, metric fluctuations can help identify the time window when an issue occurred, providing a basis for further analysis.
Logs are an important basis for identifying the cause of a failure. The cluster records detailed events during system operation, including service status, error messages, and operation records. By collecting and analyzing logs around the time of the failure, you can reconstruct the system behavior when the issue occurred and efficiently analyze the root cause.
The alert feature triggers notifications when anomalies occur during cluster operation. By reviewing alert information, operations personnel can immediately become aware of risks and, combined with the relevant failure scenario descriptions in this document, take appropriate follow-up actions.
Event audit automatically records and displays cluster events, including user events and system events:
User events: Record manual operations performed by users, such as modifying configurations and adding or removing resources.
System events: Record events automatically triggered by the system, such as virtual machine High Availability (HA) rebuild and restart.
Operations personnel can view and trace event details in the Events interface of AOC to confirm the root cause of issues, distinguish between user operations and system anomalies, and provide a basis for subsequent troubleshooting.
During troubleshooting, command-line tools can also provide useful diagnostic information. Common examples include:
zbs-node show_disk_status <disk_path>: Queries the health details of a physical disk.
zbs-tool abnormal_detect find_dead_extent: Finds dead data blocks and the virtual machines and virtual volumes affected by them.
smartctl -x <disk_path>: Collects S.M.A.R.T. information and self-test logs for a physical disk, which can be used for reporting disk failures.
······
Inspector checks cluster configurations, usage, performance, and service status to identify existing issues and potential risks in the cluster. It provides targeted solutions for each identified risk item and generates inspection reports for operations personnel to review and analyze, helping them detect existing or potential cluster issues and ensure healthy cluster operation.
