3 ms·
That is with 4-bit quantization. For practical purposes I don't see the point of running anything higher than that for inference.
by int_19h 3y ago
That is with 4-bit quantization. For practical purposes I don't see the point of running anything higher than that for inference.
- AnthonyMouse 3y agoThat's interesting though, because it implies the machine is compute-bound. A 4-bit 120B model is ~60GB, so you should get ~13 tokens/second out of 800GB/s if it was memory-bound. 7/s implies you're getting ~420GB/s. And the Max has half as many cores as the Ultra, implying it would be compute-bound too.