3 ms·
I'm waiting for GPU cards that allow the user to plug in memory modules.
by amelius 4y ago
I'm waiting for GPU cards that allow the user to plug in memory modules.
- saurik 4y agoInstead of waiting for the future maybe you could look to the past? That's how graphics cards used to work back a couple decades ago.
- TheGuyWhoCodes 4y agoAMD Radeon Pro SSG had 4 nvme slots on the card itself but that was 2017 but with direct storage API that might be able to have some gains for large models.
- buildbot 4y agoI could never get a solid answer wether that was presented as memory to the GPU or just as a PCIE switch with NVME drives hanging off one side and the GPU on another.
- TheGuyWhoCodes 4y agoAs far as I remember it was presented as a drive and was good for sequential read but you had to use AMD's API to get the full benefit
- paulmd 4y agoPCIe switch with NVMe drives and the GPU behind it, they actually just are presented straight to windows even. https://www.youtube.com/watch?v=-fEjoJO4lEM https://www.youtube.com/watch?v=-fEjoJO4lEM In principle they could be used with an API like DirectStorage RDMA or CUDA GPUDirect RDMA (which dates back to Kepler) and in this case they would never need to talk to the CPU, given appropriate software support. But it's not going to be presented as GPU memory ever, it's going to work like a block storage device you can do RDMA requests against, most likely. https://docs.nvidia.com/cuda/gpudirect-rdma/ https://docs.nvidia.com/cuda/gpudirect-rdma/ https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9563-efficient-distributed-storage-io-using-nvme-and-gpudirect-in-a-pcie-network.pdf https://developer.download.nvidia.com/video/gputechconf/gtc/... Now technically - it all depends on what you mean by "as GPU memory" because PCIe is all RDMA anyway, even CPU-to-GPU is a RDMA operation. That's why there's the whole thing about "resizable BAR" etc - that's the aperture window in CPU memory that gets mapped in from the GPU memory. So technically yes you can map those SSDs in "as GPU memory" via GPUDirect RDMA block storage (or DirectStorage), but you can do that with a regular NVMe SSD in an adapter card too. The SSG is just a "combo GPU+SSD card" in the same way QNAP makes those "combo network+SSD cards", but with a lot of fanfare/marketing around it. To be absolutely fair, Fiji/Vega is a good design for that since it doesn't have a bunch of memory packages around it. But it wasn't what AMD trumpeted it as, as the LTT video describes, it was a very specific reaction to the question of 'workstation GPUs are using a lot of memory, HBM can't be scaled as high, how do we put more memory on a Fiji/Vega GPU for workstation users". And AMD's claims that HBM meant you could just swap everything around and not have to worry about framebuffer size were never true, the PCIe bus itself is not fast enough for that. https://www.youtube.com/watch?v=NQ8YlXh-jbI&t=280s https://www.youtube.com/watch?v=NQ8YlXh-jbI&t=280s
- thrtythreeforty 4y agoThe bandwidth and latency is as much a bottleneck as the capacity is. That's why recent ML and GPU chips have moved to on-package "high-bandwidth memory." Even if you were to add more high-bandwidth memory off-package (via lots of parallel DIMMs, say), it's difficult to actually manage this memory well, because of a host of competing factors, including a) on-chip resources (controllers etc) that must consume area and power to manage the new memory, b) increased design cost for validating handling the extra (crappier than HBM) memory, c) software not knowing how to hide the (much higher) latency of this memory - deep learning isn't as good at memory hierarchies as CPUs are, d) a chip that has spare on-chip bus bandwidth for this kind of thing has wasted bus bandwidth. Et cetera. So if you are building GPUs or AI accelerators, you tend to just go ahead and build this in.
- deleted 4y ago[deleted]
- my123 4y agoNot going to happen: https://www.servethehome.com/wp-content/uploads/2022/08/NVIDIA-Grace-Bandwidth-HBM2e-DDR5-LPDDR5x-HC34.jpg https://www.servethehome.com/wp-content/uploads/2022/08/NVID... That unless an utterly revolutionary new interconnect technology comes... LPDDR is getting adopted more and more too, with CPUs losing memory expansion capabilities in exchange of huge power savings.
- amelius 4y agoI bet a lot of people would be happy if their GPU could train large models, even if it took 5x as long.
- my123 4y agoThat's getting a multi-level DRAM config, see Grace Superchip for an example. Or just a plain slower GPU: Apple's GPU series, but that's at very much higher price tags than desktop GPUs at a given perf level.