4 ms·
Time to first token, especially for smaller models, can be sharply reduced. Latency can be just as important as overall throughput, especially for inference pr
by noahbp 2mo ago
Time to first token, especially for smaller models, can be sharply reduced.
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
- fastball 2mo agoTokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
- marcelroed 2mo agoTime to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
- dingdingdang 2mo agoHave you done preliminary numbers on replacing tokenizer on, say, llama-server?
- marcelroed 2mo agoRunning the numbers now
- marcelroed 2mo agoAdded numbers here: https://news.ycombinator.com/item?id=49015014 https://news.ycombinator.com/item?id=49015014