3 ms·
Is there any info about how many tokens a second you would get with a 400b model? Without that it's like claiming a graphics card can output at 8k, but neglecti
by esperent 2y ago
Is there any info about how many tokens a second you would get with a 400b model? Without that it's like claiming a graphics card can output at 8k, but neglecting to state the frames per second. Suspect, in other words.
Outputs at 8k resolution! (+)
...at 5 fps (+)
- mmoskal 2y agoI saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take. For comparison H100 can read its memory 40 times per second, so if you use it all you can get around 40t/s. Of course in either case you don't have to fill it up, but instead use smaller model or more GPUs.
- sliken 2y agoI've seen similar claims, with nothing in particular to support it. Hopefully Nvidia clarifies this soon. AMD Strix Halo has a 128GB config with 256GB/sec. Apple has a MBP m4 max with 128GB ram for $4,700 with a 546GB/sec interface. Hopefully Nvidia can do better.