Greetings!

In my lab I’ve been testing PCIE Passthrough for GPU based workloads (Cloud Gaming, and other CUDA based workloads). With ESXI 7.0’s release there was an addition called Dynamic DirectPath IO which is an option for GPU passthrough.

I wasn’t sure what this was, so I decided to do some diving to find out.

What is Dynamic DirectPath I/O?

This screenshot is from vSphere 7 – Using Assignable Hardware with Dynamic DirectPath IO

In VMware vSphere 7.0, DirectPath IO is a feature that allows a physical PCIe device to be directly mapped to a VM, similar to Dynamic DirectPath IO. The main difference between the two is that DirectPath IO allows the VM to have direct access to the physical device all the time, while Dynamic DirectPath IO allows the VM to access the physical device only when the VM is powered on.

With DirectPath IO, the physical device is always reserved for the VM and cannot be used by the host system or other VMs. This means that the device’s capabilities are fully dedicated to the VM, which can improve performance for certain workloads. However, it also means that the device cannot be shared among multiple VMs or used by the host system, which can limit the flexibility and resource utilization of the host.

On the other hand, Dynamic DirectPath IO allows the VM to access the physical device only when the VM is powered on, and the device can be used by the host system or other VMs when it is not powered on and assigned. This allows for more flexible resource utilization and can potentially improve the overall performance of the host system. One inclusion in this is assisting and allowing DRS to operate with GPU assigned VMs. However, it also means that the VM may experience some latency when accessing the physical device, as it has to be switched from the host system to the VM.

Both DirectPath IO and Dynamic DirectPath IO can be useful in different scenarios, depending on the workload and the resources available on the host system. It is important to carefully consider the trade-offs and choose the appropriate option based on your specific needs.

Dynamic Path IO Hardware Labels

In vSphere 7.0, hardware labels are used to identify and group physical devices that can be used with Dynamic DirectPath IO. These labels allow for more flexibility when assigning hardware accelerators, such as GPUs, to virtual machines (VMs).

A PCIe device can be configured with a single hardware label, and DRS (Distributed Resource Scheduler) is responsible for initial placement of a VM configured with Dynamic DirectPath IO. The Assignable Hardware feature will then find a suitable host with an identical device, or a host with a device with an identical hardware label. This means that even if a different type of PCIe device is backing the hardware label, it can still be used by the VM.

Hardware labels are a useful tool in vSphere 7.0 for managing and optimizing the use of physical devices in a virtualization environment. They provide additional flexibility and control when assigning hardware accelerators to VMs, and help to ensure that the devices are used appropriately.

What do you need to use this feature?

There are several prerequisites that must be met in order to use Dynamic DirectPath IO in VMware vSphere 7.0:

1. Hardware compatibility: Your hardware platform must be compatible with Dynamic DirectPath IO. You can check the VMware Compatibility Guide to determine if your hardware is supported.
2. Supported devices: Only certain types of PCIe devices are supported for use with Dynamic DirectPath IO. These include GPUs, network interface cards (NICs), and storage controllers.
3. Host configuration: The host system must be configured to support Dynamic DirectPath IO. This includes enabling the feature in the vSphere Client, as well as configuring the physical device in the host BIOS and setting up the device in the vSphere Client.
4. VM configuration: The VM must be configured to use Dynamic DirectPath IO. This includes adding the physical device to the VM as a PCI device and enabling the feature in the VM settings.
5. License requirements: You must have a valid vSphere license that includes the Dynamic DirectPath IO feature.

It is important to carefully follow these prerequisites and ensure that your hardware and software are properly configured before attempting to use Dynamic DirectPath IO.

Important Links I used to write and learn this content:

vSphere 7 – Assignable Hardware – VMware vSphere Blog

vSphere VMDirectPath I/O and Dynamic DirectPath I/O: Requirements for Platforms and Devices (2142307) (vmware.com)

vSphere 7 – Using Assignable Hardware with Dynamic DirectPath IO – YouTube

What is VMware vSphere 7 Assignable Hardware? – Virtualization Howto