To set the stage: RDMA. If you haven’t already enabled it in your kernel configuration, do that now! Trust us, it’ll make a world of difference. Here’s how to enable it on Ubuntu:
# This script enables RDMA on Ubuntu by adding the necessary configuration and updating the initramfs before rebooting the system.
# Add the "options rdma_cm rdmacm" line to the rdma.conf file in the modprobe.d directory.
echo "options rdma_cm rdmacm" >> /etc/modprobe.d/rdma.conf
# Update the initramfs to include the new configuration.
update-initramfs -u
# Reboot the system to apply the changes.
reboot
Now that RDMA is enabled, memory allocation. Haswell has a nifty feature called Intel VT-D (Virtualization Technology for Directed I/O), which allows you to allocate specific memory regions directly to your DPDK workloads instead of going through the OS kernel. This can significantly reduce latency and increase performance.
To enable this, add the following lines to your `dpdk.conf` file:
# This script enables Virtualization Technology for Directed I/O (VT-d) in the DPDK configuration file.
# The following lines allocate specific memory regions directly to DPDK workloads instead of going through the OS kernel, reducing latency and increasing performance.
# Set the PCI device with address 00:19:00.0 to be associated with NUMA node 2.
pci 00:19:00.0 numa-node 2
# Set the DMA mask for the PCI device with address 00:19:00.0 to 64 bits.
pci 00:19:00.0 dma-mask 64
# Enable RDMA mode for the PCI device with address 00:19:00.0.
pci 00:19:00.0 rdma-mode on
This tells DPDK to allocate memory region `00:19:00.0` (which is the first NIC in our example) directly to NUMA node 2, and use a DMA mask of 64 bits for better performance. The `rdma-mode on` line enables VT-D mode for this device.
Next up, interrupt coalescing. Haswell supports Intel VT-C (Virtualization Technology for Credits), which allows you to group multiple interrupts together and send them as a single packet to the OS kernel. This can significantly reduce interrupt overhead and improve performance.
To enable this, add the following lines to your `dpdk.conf` file:
# Set the PCI device with address 00:19:00.0 to be associated with NUMA node 2
pci 00:19:00.0 numa-node 2
# Set the DMA mask for the PCI device to 64 bytes
pci 00:19:00.0 dma-mask 64
# Enable RDMA (Remote Direct Memory Access) mode for the PCI device
pci 00:19:00.0 rdma-mode on
# Enable VT-C (Virtualization Technology for Credits) vring (virtual ring) for the PCI device
pci 00:19:00.0 vt-c-vring-enable on
This tells DPDK to enable VT-C mode for device `00:19:00.0`.
Finally, CPU affinity. Haswell supports Intel VT-X (Virtualization Technology for Execution Control), which allows you to bind specific workloads to specific CPUs. This can improve performance by reducing context switching overhead and improving cache locality.
To enable this, add the following lines to your `dpdk.conf` file:
# This script enables Intel VT-X (Virtualization Technology for Execution Control) to improve performance by binding specific workloads to specific CPUs.
# Add the following lines to your `dpdk.conf` file to enable VT-X:
pci 00:19:00.0 numa-node 2 # Specifies the PCI device and assigns it to NUMA node 2.
pci 00:19:00.0 dma-mask 64 # Sets the DMA mask to 64 bits.
pci 00:19:00.0 rdma-mode on # Enables RDMA mode for the PCI device.
pci 00:19:00.0 vt-c-vring-enable on # Enables VT-X for the PCI device.
cpu_map 2,3 # Maps CPUs 2 and 3 to the PCI device.
This tells DPDK to bind workloads to CPUs `2` and `3`.
With these simple tweaks, your DPDK performance should be significantly improved.