Over the past decades, data centers have gone through several stages, including the rise and maturation of mainframes, standalone servers, centralized storage, virtualization, and distributed systems. In each era, data centers faced different challenges, which became the driving force for technological advancement. With further progress in technology, artificial intelligence and big data analytics have driven an explosive increase in the need for vast data storage and flexible processing, sparking new technological evolution. In this new wave, data centers continue to explore new service architectures, shifting from the traditional "compute clusters + FC SAN storage arrays" model to more flexible software-defined storage architectures that can support multiple deployment and usage scenarios.
Limitations of traditional storage architecture
In traditional storage architectures, several issues remain that require improvement:
Performance bottlenecks in storage controllers
Traditional storage systems rely on fixed-specification storage controllers to provide services. As business grows, storage controllers often face limitations due to fixed specifications and the inability to scale linearly when dealing with increasing I/O requests and capacity demands from the compute end. As the scale expands, the overall system performance actually decreases.
Tight coupling of dedicated storage hardware and software
Traditional storage architectures extensively use dedicated storage hardware from different manufacturers, with a high degree of coupling between the hardware and its control software. However, due to the lack of unified industry standards among hardware from different manufacturers, devices from different vendors are likely to be incompatible with each other. Each type of dedicated hardware has its own management mechanism and interface, significantly increasing management complexity. In addition, when new business demands arise, existing hardware may not meet these new requirements, resulting in significant investments for equipment upgrades.
As a result, traditional storage architectures have become a bottleneck in enhancing the processing capabilities of data centers. To address these challenges, software-defined storage technologies have emerged.
The rise of software-defined storage
Software-defined storage (SDS) is a distributed storage solution based on virtualization and resource pooling technologies. Its core concept is to separate the critical control and processing functions of a storage system from the underlying hardware, implementing them through software instead. Software-defined storage consolidates storage resources spread across different servers within the same cluster into a single logical storage pool and automatically creates and adjusts the required volumes based on demand. The capacity of the storage pool can also be dynamically scaled online in accordance with changes in the number of cluster servers. This flexibility enables it to meet evolving demands.
Software-defined storage allows storage services to run on general-purpose hardware without relying on expensive dedicated hardware. By shifting functions such as storage protocol conversion, data redundancy, deduplication, compression, and verification to the software layer, you can significantly reduce initial investment costs and ongoing maintenance expenses while maintaining performance. Furthermore, software-defined storage features high hardware adaptability, which allows it to run on storage hardware from various vendors and leverage hardware acceleration to enhance performance.
In terms of application scenarios, software-defined storage is particularly suited for environments requiring flexible expansion and efficient data handling, such as cloud computing, big data analytics, and AI training. In these scenarios, both existing and incremental data are rapidly growing, and the requirements for storage systems are continuously evolving. Traditional storage systems often struggle to adapt quickly to these changes. However, software-defined storage offers the advantage of easily adding new functions and optimizing existing ones through software updates and upgrades, thus avoiding the complexity of hardware replacement. In addition, software-defined distributed storage allows for hardware replacement within the cluster to enhance storage space and performance without interrupting business operations, further increasing the flexibility and adaptability of the storage system.
