4 ms·
Now this looks much more interesting! Is the top one input tokens and the second one output tokens? So 38.54 t/s on 120B? Have you tested filling the context t
by xs83 1y ago
Now this looks much more interesting! Is the top one input tokens and the second one output tokens?
So 38.54 t/s on 120B? Have you tested filling the context too?
- ggerganov 1y agoYes, I provided detailed numbers here: https://github.com/ggml-org/llama.cpp/discussions/16578 https://github.com/ggml-org/llama.cpp/discussions/16578