3 ms·
There’s so much flux in LLM requirements. 2 to 3 tokens per second was actually probably fine for most things last year. Now, with reasoning and deep searchi
by SV_BubbleTime 2y ago
There’s so much flux in LLM requirements.
2 to 3 tokens per second was actually probably fine for most things last year.
Now, with reasoning and deep searching, research models, you’re gonna generate 1000 or more tokens just as it’s talking to itself to figure out what to do for you.
So everyone’s focused on how big a model you can fit inside your ram, the inference speed is now more important than it was.
- Gracana 2y agoAbsolutely. The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need it to run ten times faster. I guess code is tough too. If I'm talking to a model I'll read everything it says, so 10-20 tok/s is well and good, but that's molasses slow if it's outputting code and I'm scanning it to see if it looks right.
- adgjlsfhk1 2y agocounterpoint: thinking models are good since they give similar quality at smaller RAM sizes. if a 16b thinking model is as good as a 60b one shot model, you can use more compute without as much RAM bottleneck
- terribleperson 2y agoCounter-counterpoint: RAM costs are coming down fast this year. Compute, not so much. I still agree, though.