4 ms·
> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link So you have a full terabyte per second of bandwidth? What GPU is that? (The 64GB/s number is
by Dylan16807 2y ago
> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link
So you have a full terabyte per second of bandwidth? What GPU is that?
(The 64GB/s number is an x4 link. If you meant you have over four times that, then it sounds like CXL would be pretty competitive.)
- fc417fc802 2y agoRTX 4090 comes to mind. Dunno that I'd consider that a "years old pleb tier non-HBM GPU" though.
- schmidtleonard 2y agohttps://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 Memory Size: 24 GB Memory Type: GDDR6X Memory Bus: 384 bit Bandwidth: 1.01 TB/s Bandwidth between where the LLM is stored and where your matrix*vector multiplies are done is the important figure for inference. You want to measure this in terabytes per second, not gigabytes per second. A 7900XTX also has 1TB/s on paper, but you'll need awkward workarounds every time you want to do something (see: article) and half of your workloads will stop dead with driver crashes and you need to decide if that's worth $500 to you. Stacking 3090s is the move if you want to pinch pennies. They have 24GB of memory and 936GB/s of bandwidth each, so almost as good as the 4090, but they're as cheap as the 7900XTX with none of the problems. They aren't as good for gaming or training workloads, but for local inference 3090 is king. It's not a coincidence that the article lists the same 3 cards. These are the 3 cards you should decide between for local LLM, and these are the 3 cards a true competitor should aim to exceed.
- Dylan16807 2y agoA 4090 is not "years old pleb tier". Same for 3090 and 7900XTX. There's a serious gap between CXL and RAM, but it's not nearly as big as it used to be.
- adrian_b 2y agoAlready an ancient Radeon VII from 5 years ago had 1 terabyte per second of memory bandwidth. Later consumer GPUs have regressed and only RTX 4090 offers the same memory bandwidth in the current NVIDIA generation.
- Dylan16807 2y agoRadeon VII had HBM. So I can understand a call for returning to HBM, but it's an expensive choice and doesn't fit the description.
- ryao 2y agoThat seems unlikely given that the full HBM supply for the next year has been earmarked for enterprise GPUs. That said, it would be definitely nice if HBM became available for consumer GPUs.
- ryao 2y agoThe 3090 Ti and 4090 both have 1.01TB/sec memory bandwidth: https://www.techpowerup.com/gpu-specs/geforce-rtx-3090-ti.c3829 https://www.techpowerup.com/gpu-specs/geforce-rtx-3090-ti.c3...
- Dylan16807 2y agoBut as I addressed earlier, those are not "years old pleb tier".