>What’s a useful amount of tokens/second? Depends entirely on your application. If you want a real time chatbot, anything more than 10 tokens/s is probably gen

>What’s a useful amount of tokens/second? Depends entirely on your application. If you want a real time chatbot, anything more than 10 tokens/s is probably generating text faster than the user can read, so it's fine. Do you want code suggestions or completions? Probabaly also fine, but a bit slow. Do you want to create summaries for 1000 documents? That's gonna be really slow. But at this frontier of the field, tokens/s is not really the issue. Performance vs. quality is. If you lose a whole lot of accuracy by quantizing floats down to single digit precisions but in turn are able to run 70B parameter models, you often still get better results than less thoroughly quantized 7B parameter models. It's all about getting big models to run on memory limited GPUs.