3 ms·
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per
by marcelroed 4mo ago
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
- dingdingdang 4mo agoHave you done preliminary numbers on replacing tokenizer on, say, llama-server?
- marcelroed 4mo agoRunning the numbers now
- marcelroed 4mo agoAdded numbers here: https://news.ycombinator.com/item?id=49015014 https://news.ycombinator.com/item?id=49015014