3 ms·
What’s the definition of “prompt processing” vs “token generation”? Is that separately comparing the time it takes to preprocess the input prompt (prompt_lengt
by cgearhart 3y ago
What’s the definition of “prompt processing” vs “token generation”?
Is that separately comparing the time it takes to preprocess the input prompt (prompt_length / pp_token_rate = time_to_first_token) and then the token generation rate is the time for each successive token?
I also see something about bs batch size. Is batching relevant for a locally run model? (Usually you only have one prompt at a time, right?)
- discobot 3y agoregarding the prompt processing and token generation you are correct it makes sense to benchmark them infependently since prompt processing is done in parralel for each token and is compute bound and token generation is sequential and bound by memory banwidth
- idonotknowwhy 3y agoYeah prompt processing is tokenizing the input, which includes the system prompt, chat history and your chat text. Generation is the response it sends back
- andy99 3y agoIt says at the top that prompts are processed with a batch size of 512, while new tokens can only be generated one at a time with batch size of 1. Prompt processing doesn't actually need to compute logits for each new token, just cache the KV values and so is much faster than actual inference.
- deleted 3y ago[deleted]