3 ms·

If I understand it correctly, at least for nVidia cards, when the GPU is running code in kernel space it is capable of going around the IOMMU and access physica

If I understand it correctly, at least for nVidia cards, when the GPU is running code in kernel space it is capable of going around the IOMMU and access physical addresses directly. This is required to make stuff like GPUdirect RDMA over Infiniband work.

the8472 12y ago
>GPU is running code in kernel space I'm not up to speed with GPU-computing. Do you mean the GPU drivers executing in kernel mode or is there some magic going on in the GPU itself that it has non-MMU'd access to the main memory? > This is required to make stuff like GPUdirect RDMA over Infiniband work. Well, let's assume that many people don't need this. Would IOMMU protection apply then?
- semi-extrinsic 12y ago
  To answer your first question: I don't think there's any code running on the GPU in kernel space except for the DRM driver part of the GPU drivers. As for the second question, I'm not sure. nVidia at least uses the same driver for GTX and Tesla class cards, but I assume the driver runs different code based on what card is installed, and presumably turns features like IOMMU on/off based on the card present. Whether SLI also requires working around IOMMU I don't know.
  - the8472 12y ago
    > and presumably turns features like IOMMU on/off based on the card present. Presumably. Yeah. That's why I asked in the first place ;)
  - wtallis 12y ago
    I doubt that they could disable the IOMMU based on the presence of a pro card, since the virtualization uses that require an IOMMU are a lot more common than anything RDMA that might conflict with an IOMMU.
    - semi-extrinsic 12y ago
      Per the nVidia documentation: RDMA for GPUDirect currently relies upon all physical addresses being the same from the PCI devices' point of view. This makes it incompatible with IOMMUs and hence they must be disabled for RDMA for GPUDirect to work. Sounds like they require the user to disable it.
      - wtallis 12y ago
        So the real requirement is probably just that the GPU and the NIC get compatible mappings, and you could probably compartmentalize them as a pair—the same as how all the legacy PCI devices behind a PCIe to PCI bridge need to dealt with as a unit for IOMMU purposes.

the8472 12y ago

>GPU is running code in kernel space I'm not up to speed with GPU-computing. Do you mean the GPU drivers executing in kernel mode or is there some magic going on in the GPU itself that it has non-MMU'd access to the main memory? > This is required to make stuff like GPUdirect RDMA over Infiniband work. Well, let's assume that many people don't need this. Would IOMMU protection apply then?

semi-extrinsic 12y ago
To answer your first question: I don't think there's any code running on the GPU in kernel space except for the DRM driver part of the GPU drivers. As for the second question, I'm not sure. nVidia at least uses the same driver for GTX and Tesla class cards, but I assume the driver runs different code based on what card is installed, and presumably turns features like IOMMU on/off based on the card present. Whether SLI also requires working around IOMMU I don't know.
- the8472 12y ago
  > and presumably turns features like IOMMU on/off based on the card present. Presumably. Yeah. That's why I asked in the first place ;)
- wtallis 12y ago
  I doubt that they could disable the IOMMU based on the presence of a pro card, since the virtualization uses that require an IOMMU are a lot more common than anything RDMA that might conflict with an IOMMU.
  - semi-extrinsic 12y ago
    Per the nVidia documentation: RDMA for GPUDirect currently relies upon all physical addresses being the same from the PCI devices' point of view. This makes it incompatible with IOMMUs and hence they must be disabled for RDMA for GPUDirect to work. Sounds like they require the user to disable it.
    - wtallis 12y ago
      So the real requirement is probably just that the GPU and the NIC get compatible mappings, and you could probably compartmentalize them as a pair—the same as how all the legacy PCI devices behind a PCIe to PCI bridge need to dealt with as a unit for IOMMU purposes.