3 ms·
Prefill is 400 t/s in that hardware. Just if the prompt is very short you can't see the real speed and it will default to single token context processing.
by antirez 5mo ago
Prefill is 400 t/s in that hardware. Just if the prompt is very short you can't see the real speed and it will default to single token context processing.
- simonw 5mo agoHah, that's my fault for just using "Generate an SVG of a pelican riding a bicycle" as my test prompt!