RDMA (Remote Direct Memory Access) is a technology that can bypass the operating system kernels of both endpoints and directly access memory data over the network. Because RDMA does not pass through the operating system, using the RDMA protocol can save a large amount of CPU resources, improve system throughput, and reduce network communication latency.
ACOS clusters support enabling RDMA for the storage network. Remote replica transmission can bypass multiple OS buffers, reducing latency, improving overall cluster performance, and improving both latency and throughput.
Using RDMA can bring the following advantages:
Under high bandwidth pressure, CPU usage is extremely low. When high-speed networks of 25 GbE or higher, such as 25 GbE, 100 GbE, and 200 GbE, are used, RDMA can fully utilize their bandwidth advantages.
RDMA avoids operating system buffer overhead and implements direct communication of the distributed block storage service between different nodes, effectively reducing latency.
The deployment architecture is the same as that in non-RDMA scenarios. ACOS directly controls physical network ports. The configuration form is that physical network ports are added to a virtual distributed switch, and the storage network IP address is configured on the virtual interface of the virtual distributed switch.
On the VMware ESXi platform, ACOS in the SCVM cannot directly control network ports. Therefore, after RDMA is enabled, SR-IOV must be enabled for physical network ports. ESXi uses the Physical Function (PF) of the network port and provides Virtual Functions (VFs) to the SCVM. When a physical network port supports RDMA, its VFs also have RDMA capabilities. This mode is close to passthrough, and its performance is also close to direct passthrough of the entire NIC.
If the storage network has multiple physical network ports, multiple VFs are allocated to the SCVM, and multilink processing is required inside the SCVM. Therefore, one OVS layer is added inside the SCVM. To allow the OVS to communicate with the outside, an independent vSwitch must be added and Promiscuous mode must be enabled.
In the SCVM, the SR-IOV VF network serves as the storage network and is responsible for data transmission between nodes. The original network of the vmxnet3 type serves as the VMware access network and is responsible for communication between the SCVM and the local ESXi host.
After RDMA is enabled, the network architecture changes as follows:
Management network (Management)
As in the original mode, this network connects to a 1 GbE physical network port.
NFS network
As in the original mode, this network is not connected to any physical network port and is used only for communication between the SCVM and ESXi.
VMware access network
This network is used for communication between the SCVM and the local ESXi host. Traffic is directly forwarded through the vSwitch and does not pass through physical network ports.
Storage network (RDMA)
This network is used for communication between the SCVM and external devices, such as other ESXi hosts or SCVMs.
The vSwitch in this network is not connected to physical network ports and is used only for SR-IOV configuration. Promiscuous mode must be enabled. Directly configure the SR-IOV VFs in the RDMA port group under this vSwitch. Traffic does not pass through the vSwitch, as indicated by dashed lines, but is sent and received through the VFs and directly passes through the physical switch.
After the network architecture is adjusted, the RDMA network uses the original storage IP address, and a VMware access network IP address must be added for the network. When the SCVM communicates externally, if the destination is the local ESXi host, the VMware access network IP address is used for sending. In other cases, the storage network IP address is used for sending.
