Erasure coding (EC) is a data redundancy technology designed to reduce the space amplification issues inherent in the replication policy. Suppose a data block has a size of N bytes. An erasure coding policy splits this block into K equal-sized data shards and generates M parity shards through specific encoding operations, resulting in a total of K+M data shards. In these K+M data shards, as long as the lost shards do not exceed M, the original data can be recovered from the remaining K data shards.
The space amplification ratio for erasure coding is (K+M)/K, where M is generally less than K, resulting in a space amplification ratio of less than 2. Compared to the replication mode (such as a space amplification rate of 3 for 3 replicas), erasure coding offers higher space utilization efficiency. For example, a 4+2 erasure coding configuration can provide fault tolerance equivalent to 3 replicas while occupying only 1.5 times the original data size.
The erasure coding policy of ABS supports multiple K+M combinations. It can tolerate up to 4 data disk failures within the cluster without affecting normal data access.
Compared to replication policy, when using the erasure coding policy, updating data blocks involves updating parity shards, especially when I/O requests differ in size from data shards, necessitating partial data updates. Typically, this requires reading the entire shard data for complete computation, which can impact performance. Therefore, the erasure coding policy is available only in a tiered deployment mode. In the tiered deployment mode, all data update requests are initially written into the cache tier using replication. They are then organized and merged within the cache tier before being written to the capacity tier in the form of erasure coding. In this mode, the cache tier is used only for caching, while the capacity tier always stores data using erasure coding. This ensures the write performance of erasure coded volumes.
To ensure data reliability, ABS regularly checks data shards stored using erasure coding and proactively initiates repair mechanisms when anomalies are detected. Unlike the replication mode, erasure coding repair requires reading data from multiple data shards for rebuild calculations, rather than relying on a single complete replica.
