3 ms·
What? Below 100tk/s is worthless? I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright f
by wolvoleo 2mo ago
What? Below 100tk/s is worthless?
I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace.
Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally.
And will it remain free? How are they going to make back all those trillions of investment?
But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
- zaj00l 2mo agoI wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing. Still, as you said - for day to day, 50-60 tokens / s is a good baseline.