9 ms·

> as modern dedicated GPUs, can theoretically do zero-copy transfers by mapping the appropriate memory on both the CPU and GPU. Is this true for dgpus? How doe

> as modern dedicated GPUs, can theoretically do zero-copy transfers by mapping the appropriate memory on both the CPU and GPU. Is this true for dgpus? How does this work?

yaantc 4y ago
This is not specific to dGPU, it could apply to any PCIe device. Emphasis on "theoretically" too. On the device (dGPU here), it is possible to route memory accesses to part of the internal address space to the PCIe controller. In turn, the PCIe controller can translate such received memory access into a PCIe request (read or write), in the different PCIe address space, with some address translation. This PCIe request goes to the PCIe host (CPU in a dGPU scenario). Here too the host PCIe controller can map the PCIe request, using using a PCIe address space address, into the host address space. And this can go to the host memory (after IOMMU filtering and address translation usually). And all this back for the return trip to the device in case of a read. So latency would be rather high, but technically possible. In most application such transfers are offloaded to a DMA in the PCIe controller doing a copy between PCIe and local address spaces, but a processing core can certainly do a direct access without DMA if all the address mappings are suitably configured.
- alanfranz 4y ago
  Uuuuuh, ok, but.. what’s the point of doing so? If I do zero-copy on a shared memory area between cpu and gpu, the advantage is clear - no copy and fast transfer. If I map some host memory to the GPU… I get worse latency and worse bandwidth. Most likely not a win.
  - yaantc 4y ago
    That's why the author says "theoretically" I guess ;) Yes in practice you probably wouldn't want your GPU compute engines to do such direct accesses and stall for a long time on each access, even for a one-shot streaming processing. Then even to avoid using the GPU main memory one would likely use DMA copies to a local working memory and do the processing there by chunks. But the direct mapping can still be convenient: a local DMA engine (or any HW coprocessor) can access host or GPU memory in the same way.
kevingadd 4y ago
In theory for a long time you've been able to "persistently map" A GPU side buffer that houses things like indexes, vertex data, or even textures, and then write directly* into GPU memory from the CPU without a staging buffer. This was referred to as 'AZDO' (Approaching Zero Driver Overhead) in the OpenGL space and eventually fed into the design of Vulkan and Direct3D 12 (see https://www.gdcvault.com/play/1020791/Approaching-Zero-Driver-Overhead-in https://www.gdcvault.com/play/1020791/Approaching-Zero-Drive... if you're curious about all of this) I say in theory and used an asterisk because I think it's generally the case that the driver could lie and just maintain an illusion for you by flushing a staging buffer at the 'right time'. But in practice my understanding is that the memory writes will go straight over the PCIe bus to the GPU and into its memory, perhaps with a bit of write-caching/write-combining locally on the CPU. It would be wise to make sure you never read from that mapped memory :)
- garaetjjte 4y ago
  AZDO is general term for techniques that reduce driver overhead, not limited to persistent mapping. OpenGL drivers have a habit to try to second-guess the application (though this depends on the driver, eg. Nvidia guesses a lot, Mesa not so much), but passing/not-passing GL_CLIENT_STORAGE_BIT to glBufferStorage should decide whether buffer should reside in CPU/GPU side memory. In D3D11 directly mapped GPU memory is known as D3D11_USAGE_DYNAMIC.
LargoLasskhyfv 4y ago
I think that was the whole point of https://en.wikipedia.org/wiki/Heterogeneous_System_Architecture https://en.wikipedia.org/wiki/Heterogeneous_System_Architect... by https://github.com/HSAFoundation/HSA-Runtime-AMD https://github.com/HSAFoundation/HSA-Runtime-AMD , as a means to standardise ways to access. Not much came out of it, it seems. AMD itself seems to have a aboned that particular thing. Seems to be that there are other ways now, to achieve this, on lower levels in the hardware, 'transparent' to the layers above. 3D-Vcache stacked on top of the die, and https://www.techarp.com/computer/amd-infinity-cache-explained/ https://www.techarp.com/computer/amd-infinity-cache-explaine... comes to mind.
- Salgat 4y ago
  The problem is that the only PoC was on APUs and the PS4, both of which had shared access to the RAM anyways making it much simpler to implement. Same is true for M1. The real trick is doing it on something that doesn't have unified memory.
diamondlovesyou 4y ago
See AMD "Smart Memory" a.k.a. PCIe Large Bar. This expands the amount of GPU memory that the CPU can directly access, usually to the GPU's entire memory range (ordinarily only ~256Mb is accessible). GPU->CPU Reads have very high latencies, but that's not an issue for CPU->GPU writes. GPUs have been able to access "host" memory for a long time now, with a few restrictions: you have to setup the GPU mappings first and pin the pages in memory.

> as modern dedicated GPUs, can theoretically do zero-copy transfers by mapping the appropriate memory on both the CPU and GPU. Is this true for dgpus? How does this work?

yaantc 4y ago

This is not specific to dGPU, it could apply to any PCIe device. Emphasis on "theoretically" too. On the device (dGPU here), it is possible to route memory accesses to part of the internal address space to the PCIe controller. In turn, the PCIe controller can translate such received memory access into a PCIe request (read or write), in the different PCIe address space, with some address translation. This PCIe request goes to the PCIe host (CPU in a dGPU scenario). Here too the host PCIe controller can map the PCIe request, using using a PCIe address space address, into the host address space. And this can go to the host memory (after IOMMU filtering and address translation usually). And all this back for the return trip to the device in case of a read. So latency would be rather high, but technically possible. In most application such transfers are offloaded to a DMA in the PCIe controller doing a copy between PCIe and local address spaces, but a processing core can certainly do a direct access without DMA if all the address mappings are suitably configured.

alanfranz 4y ago
Uuuuuh, ok, but.. what’s the point of doing so? If I do zero-copy on a shared memory area between cpu and gpu, the advantage is clear - no copy and fast transfer. If I map some host memory to the GPU… I get worse latency and worse bandwidth. Most likely not a win.
- yaantc 4y ago
  That's why the author says "theoretically" I guess ;) Yes in practice you probably wouldn't want your GPU compute engines to do such direct accesses and stall for a long time on each access, even for a one-shot streaming processing. Then even to avoid using the GPU main memory one would likely use DMA copies to a local working memory and do the processing there by chunks. But the direct mapping can still be convenient: a local DMA engine (or any HW coprocessor) can access host or GPU memory in the same way.

kevingadd 4y ago

In theory for a long time you've been able to "persistently map" A GPU side buffer that houses things like indexes, vertex data, or even textures, and then write directly* into GPU memory from the CPU without a staging buffer. This was referred to as 'AZDO' (Approaching Zero Driver Overhead) in the OpenGL space and eventually fed into the design of Vulkan and Direct3D 12 (see https://www.gdcvault.com/play/1020791/Approaching-Zero-Driver-Overhead-in https://www.gdcvault.com/play/1020791/Approaching-Zero-Drive... if you're curious about all of this) I say in theory and used an asterisk because I think it's generally the case that the driver could lie and just maintain an illusion for you by flushing a staging buffer at the 'right time'. But in practice my understanding is that the memory writes will go straight over the PCIe bus to the GPU and into its memory, perhaps with a bit of write-caching/write-combining locally on the CPU. It would be wise to make sure you never read from that mapped memory :)

garaetjjte 4y ago
AZDO is general term for techniques that reduce driver overhead, not limited to persistent mapping. OpenGL drivers have a habit to try to second-guess the application (though this depends on the driver, eg. Nvidia guesses a lot, Mesa not so much), but passing/not-passing GL_CLIENT_STORAGE_BIT to glBufferStorage should decide whether buffer should reside in CPU/GPU side memory. In D3D11 directly mapped GPU memory is known as D3D11_USAGE_DYNAMIC.

LargoLasskhyfv 4y ago

I think that was the whole point of https://en.wikipedia.org/wiki/Heterogeneous_System_Architecture https://en.wikipedia.org/wiki/Heterogeneous_System_Architect... by https://github.com/HSAFoundation/HSA-Runtime-AMD https://github.com/HSAFoundation/HSA-Runtime-AMD , as a means to standardise ways to access. Not much came out of it, it seems. AMD itself seems to have a aboned that particular thing. Seems to be that there are other ways now, to achieve this, on lower levels in the hardware, 'transparent' to the layers above. 3D-Vcache stacked on top of the die, and https://www.techarp.com/computer/amd-infinity-cache-explained/ https://www.techarp.com/computer/amd-infinity-cache-explaine... comes to mind.

Salgat 4y ago
The problem is that the only PoC was on APUs and the PS4, both of which had shared access to the RAM anyways making it much simpler to implement. Same is true for M1. The real trick is doing it on something that doesn't have unified memory.

diamondlovesyou 4y ago

See AMD "Smart Memory" a.k.a. PCIe Large Bar. This expands the amount of GPU memory that the CPU can directly access, usually to the GPU's entire memory range (ordinarily only ~256Mb is accessible). GPU->CPU Reads have very high latencies, but that's not an issue for CPU->GPU writes. GPUs have been able to access "host" memory for a long time now, with a few restrictions: you have to setup the GPU mappings first and pin the pages in memory.