The upper limit of I/O performance for a cluster varies depending on the software version and hardware configuration. When the read bandwidth, write bandwidth, or total bandwidth of a host continuously approaches or has exceeded the threshold, the system triggers an I/O bandwidth alert to indicate a potential performance bottleneck risk.
To ensure stable operations, it is recommended to run stress tests in the cluster using a performance testing platform before the workload goes live, and set the alert threshold for daily monitoring to approximately 70% of the measured maximum I/O capacity.
Total I/O bandwidth for host { .labels.hostname } exceeds { .threshold }.
The read bandwidth for host { .labels.hostname } exceeds { .threshold }.
The write bandwidth for host { .labels.hostname } exceeds { .threshold }.
The total I/O bandwidth of the physical disk pool {.id} on the host { .labels.hostname } exceeds { .threshold }.
The read bandwidth of the physical disk pool {.id} on the host { .labels.hostname } exceeds { .threshold }.
The write bandwidth of the physical disk pool {.id} on the host { .labels.hostname } exceeds { .threshold }.
The host may experience a performance bottleneck. Virtual machines on the host and external compute clients using the host's storage resources may experience slower response times or increased latency due to storage capacity constraints.
The current I/O traffic on the host has exceeded the configured alert threshold for more than 5 minutes.
Scenario 1: Expected high load (no action required)
If the alert is caused by cluster stress testing, planned high-load workloads, or similar events where bandwidth peaks are brief and services are running normally, you can dismiss the alert without taking any action.
Scenario 2: Sustained high load (scale-out recommended)
If workloads consistently consume the maximum bandwidth capacity of a node, it is recommended to add nodes or upgrade the network, physical disk, or other hardware configuration to increase the maximum throughput capacity of the node. This ensures performance redundancy to handle sudden load spikes and maintain stable performance in the event of a failure.
Scenario 3: Unexpected high bandwidth (investigation required)
If there are no workloads expected to consume high bandwidth, use AOC or the Observability platform to check the top-N traffic view for virtual machines and determine whether any single object has experienced abnormally high traffic.
If so, check whether the applications on the virtual machine or associated virtual volumes have abnormal tasks generating unexpected traffic. If unexpected traffic is identified, it is recommended to first configure a throttling policy for the target virtual machine or virtual volume in AOC to avoid affecting other normal operations, and then investigate the specific cause of the abnormal traffic.
Scenario 4: Uneven node load (hot migration recommended)
If the traffic is evenly distributed across different virtual machines or virtual volumes on the alerted node, and other nodes in the cluster are under relatively light load, consider using hot migration to move some virtual machines to less loaded nodes.
Note that because data in physical disks does not move immediately, migrating virtual machines or other actions that change data paths may initially cause a slight performance decrease for the migrated virtual machines. However, as running data gradually accumulates on the new node, the overall cluster performance will improve compared to when the load was concentrated on a single host.
