Modern "consumer" GPUs have relatively narrow busses, which is a deliberate design choice. But its not about throughput, its about cost and power savings in gam

Modern "consumer" GPUs have relatively narrow busses, which is a deliberate design choice. But its not about throughput, its about cost and power savings in gaming workloads. Old GPUs like the 290X had a 512 bit bus (like the M1 Max) and 16 GDDRX ICs. Actual ML GPUs transitioned to HBM, but the capacity is relatively limited. I think Nvidia thought 96GB per GPU would be plenty when they were planning things out years ago, but the explosion of model sizes seems to have taken all the hardware manufacturers by surprise.