3 ms·

A PCIE 4.0 16x link should provide around 32GB/s of bandwidth, close to the 33GB/s of DDR5-3200. In a perfect world, it would seem to me that doing 100% offload

A PCIE 4.0 16x link should provide around 32GB/s of bandwidth, close to the 33GB/s of DDR5-3200. In a perfect world, it would seem to me that doing 100% offloading (streaming everything as needed from system memory) should be equivalent to doing the calculations in system memory in the first place. The GPU memory just acts as a cache, and should only speed up the processing.

BobbyJo 3y ago
The full mem-swap congo line is DRAM<->PCIE<->VRAM<->GPU. PCIE is the weak link, but I'd be willing to bet that those transfers are't 100% overlapped, and PCIE transfer rate represents best case speed as opposed to expected. In the case of unidirectional writes, you'd have to cut that speed in half.
dragontamer 3y ago
Every modern motherboard is dual-channel DDR5 at a minimum, maybe quad-channel. 2x sticks of 32GB/s RAM, properly configured, will run at 64GB/s of bandwidth. Modern servers are quad, hex, or oct-channel (4x, 6x, or 8x parallel sticks of RAM) in practice. Or even more (ex: Intel Xeon Platinums are 6x channel per socket, so an 8x socket 8x CPU Xeon Platinum will be like 48x DDR5 parallel RAM sticks). ---------- PCIe x16 by the way, is 16x parallel lanes of PCIe. All the parallelism is already innate in modern systems. --------- L3 cache is TB/s bandwidth IIRC. CPUs inside of CPU-space will automatically be caching a lot of those RAM commands, so you can go above the RAM-bandwidth limitations in practice, though it depends on how your code accesses RAM. GPUs have very small caches, and have higher latency to those caches. Instead, GPUs rely upon register-space and extreme amounts of SMT/wavefronts to kinda-sorta hyperthread their cores to hide all that latency.
gautamcgoel 3y ago
This argument makes sense only if you assume that your model is too large to fit in the GPU's RAM and hence has to reside mainly in the CPU's RAM.
- delusional 3y ago
  That's true. The comment i was responding to talked about offloading. I was assuming he was talking about offloading part of the core model to the system RAM which would need to be reloaded frequently.

BobbyJo 3y ago

The full mem-swap congo line is DRAM<->PCIE<->VRAM<->GPU. PCIE is the weak link, but I'd be willing to bet that those transfers are't 100% overlapped, and PCIE transfer rate represents best case speed as opposed to expected. In the case of unidirectional writes, you'd have to cut that speed in half.

dragontamer 3y ago

Every modern motherboard is dual-channel DDR5 at a minimum, maybe quad-channel. 2x sticks of 32GB/s RAM, properly configured, will run at 64GB/s of bandwidth. Modern servers are quad, hex, or oct-channel (4x, 6x, or 8x parallel sticks of RAM) in practice. Or even more (ex: Intel Xeon Platinums are 6x channel per socket, so an 8x socket 8x CPU Xeon Platinum will be like 48x DDR5 parallel RAM sticks). ---------- PCIe x16 by the way, is 16x parallel lanes of PCIe. All the parallelism is already innate in modern systems. --------- L3 cache is TB/s bandwidth IIRC. CPUs inside of CPU-space will automatically be caching a lot of those RAM commands, so you can go above the RAM-bandwidth limitations in practice, though it depends on how your code accesses RAM. GPUs have very small caches, and have higher latency to those caches. Instead, GPUs rely upon register-space and extreme amounts of SMT/wavefronts to kinda-sorta hyperthread their cores to hide all that latency.

gautamcgoel 3y ago

This argument makes sense only if you assume that your model is too large to fit in the GPU's RAM and hence has to reside mainly in the CPU's RAM.

delusional 3y ago
That's true. The comment i was responding to talked about offloading. I was assuming he was talking about offloading part of the core model to the system RAM which would need to be reloaded frequently.