The alert is triggered when the write cache or hot data space utilization exceeds the configured threshold.
The write cache usage of the cluster exceeds { .threshold }.
Host { .labels.name }: Hot data has used more than { .threshold } of the total cache size of this host for over 12 hours.
The hot data in the physical disk pool {.id} of the host { .labels.name } exceeds { .threshold } of the total cache space, and this condition has persisted for over 12 hours.
Insufficient write cache space may cause a decline in cluster I/O performance:
Write requests that can be written directly to the capacity tier (large-block aligned writes of 256 KiB under medium load, and 8 KiB-aligned replica-mode writes under high load) will bypass the write cache and write directly to the capacity tier. When the cache tier significantly outperforms the capacity tier, cluster performance will decline.
When the alert is triggered, the cluster has very likely already entered a high-load state caused by high write cache space utilization. To ensure cache space remains available for I/O that must go through the write cache (erasure coding writes and 8 KiB unaligned replica writes), the system simultaneously enables accelerated cache flushing and workload throttling. At this point, the system resources available to the workload will be noticeably lower than under low-load conditions, resulting in a significant performance decline.
Common causes include:
A large-capacity cache disk in the cluster is experiencing an anomaly, or the node hosting it is experiencing an anomaly.
High-intensity random write traffic is being generated for large-capacity volumes. This is more likely to occur when large-capacity volumes with high write bandwidth coexist with relatively frequent snapshot schedules.
Check for hardware failures or failures that have caused some storage services to become unavailable. If any, fix the failures first.
Use the monitoring feature to check whether there was unexpected high-volume write traffic before the alert was triggered. If so, apply throttling to the affected virtual machines or virtual volumes, or shut down the relevant virtual machines to reduce the current cluster load.
If workload demands persistently exceed the cluster's hardware capacity, take the following measures:
If the burst traffic volume is fixed and the I/O load on the capacity tier remains low during other periods, add cache disk space to accommodate the burst traffic.
If the burst traffic workload has lower priority, apply ongoing throttling to reduce the impact on other higher-priority workloads in the cluster.
If the high-volume, large-capacity workload has high priority and is sensitive to throughput, consider migrating the workload to an all-flash cluster.
