4 ms·
Oh, sorry, I assumed the 8 was for quantization. 8x7b is a new syntax for me. Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?
by SeanAnderson 3y ago
Oh, sorry, I assumed the 8 was for quantization. 8x7b is a new syntax for me.
Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?
- tome 3y agoI guess that's total throughput, rather than per user? You can increase total throughput by scaling horizontally. You can't increase throughput per user that way.