2 ms·
I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give o
by mmoskal 2y ago
I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take.
For comparison H100 can read its memory 40 times per second, so if you use it all you can get around 40t/s.
Of course in either case you don't have to fill it up, but instead use smaller model or more GPUs.
- sliken 2y agoI've seen similar claims, with nothing in particular to support it. Hopefully Nvidia clarifies this soon. AMD Strix Halo has a 128GB config with 256GB/sec. Apple has a MBP m4 max with 128GB ram for $4,700 with a 546GB/sec interface. Hopefully Nvidia can do better.