3 ms·
Absolutely. The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need
by Gracana 2y ago
Absolutely.
The thinking models really hurt. I was happy with anything that ran at least as fast as I could read, then "thinking" became a thing and now I need it to run ten times faster.
I guess code is tough too. If I'm talking to a model I'll read everything it says, so 10-20 tok/s is well and good, but that's molasses slow if it's outputting code and I'm scanning it to see if it looks right.
- adgjlsfhk1 2y agocounterpoint: thinking models are good since they give similar quality at smaller RAM sizes. if a 16b thinking model is as good as a 60b one shot model, you can use more compute without as much RAM bottleneck
- terribleperson 2y agoCounter-counterpoint: RAM costs are coming down fast this year. Compute, not so much. I still agree, though.