4 ms·
I'm a bit rusty on the details, but I think the general reason that GPUs have less memory than you want is because they are typically optimized for throughput,
by kmod 4y ago
I'm a bit rusty on the details, but I think the general reason that GPUs have less memory than you want is because they are typically optimized for throughput, which limits capacity in a couple ways. One is higher power consumption, and another is low max distance between processors and memory.
We already have the ability to put far more memory in GPUs than they currently have, but we don't because it hurts throughput. I don't think technical advances will change the existence of a throughput/capacity tradeoff, but I do think it's an interesting idea that maybe DL models benefit from a different tradeoff than graphics applications. Personally I'd guess that's a larger factor here than incremental changes in capabilities, because the two types of memory are both improving at the same time. But who knows! Maybe the technologies will converge enough that it will no longer make sense to have two different variants, though I wouldn't bet on it.
- brucethemoose2 4y agoModern "consumer" GPUs have relatively narrow busses, which is a deliberate design choice. But its not about throughput, its about cost and power savings in gaming workloads. Old GPUs like the 290X had a 512 bit bus (like the M1 Max) and 16 GDDRX ICs. Actual ML GPUs transitioned to HBM, but the capacity is relatively limited. I think Nvidia thought 96GB per GPU would be plenty when they were planning things out years ago, but the explosion of model sizes seems to have taken all the hardware manufacturers by surprise.
- Osiris 4y agoI have a VEGA 56 with HBM and a 2048 bit bus. It gets crushed by newer cards with far less bandwidth. It was great for crypto mining though.
- brucethemoose2 4y agoYeah I am not saying older cards were better, just that they were capable of holding much more VRAM with no silicon changes.