3 ms·
What’s a useful amount of tokens/second? What’s a speed like ChatGPT 4? I’m trying to understand if there are agreed upon metrics such as “frames per second”
by yieldcrv 3y ago
What’s a useful amount of tokens/second?
What’s a speed like ChatGPT 4?
I’m trying to understand if there are agreed upon metrics such as “frames per second” in other fields or niches
- brucethemoose2 3y agoMore than ~10 tokens/s streamed feels OK to read. I can kinda live with 4-5 tokens/s (which is what I get on llama 70B on my 3090 with llama.cpp CPU offload). What feels "good" depends on the person and the type of content, but 35 tokens/s should feel very fast.
- enjeyw 3y agoI wonder if anyone will end up employing the "human" solution of padding sentences out with filler words like 'Ummmm' or 'So like' while the generator is determining the actual next token...
- visarga 3y agoOnce the model finished reading the prompt and starts answering tokens are generated at constant speed, there are no Ummm... gaps. But some tokens are already decided from the first layers and other tokens need the full depth. So there is a notion of "thinking harder" on particular tokens. We could detect token difficulty and fake the output to show the pauses.
- sigmoid10 3y ago>What’s a useful amount of tokens/second? Depends entirely on your application. If you want a real time chatbot, anything more than 10 tokens/s is probably generating text faster than the user can read, so it's fine. Do you want code suggestions or completions? Probabaly also fine, but a bit slow. Do you want to create summaries for 1000 documents? That's gonna be really slow. But at this frontier of the field, tokens/s is not really the issue. Performance vs. quality is. If you lose a whole lot of accuracy by quantizing floats down to single digit precisions but in turn are able to run 70B parameter models, you often still get better results than less thoroughly quantized 7B parameter models. It's all about getting big models to run on memory limited GPUs.