4 ms·

No you don't; it's often overkill to use the SOTA models. People want SOTA because it's shiny, but there are a lot of tasks where it's cheaper and more efficien

by hk__2 3mo ago

No you don't; it's often overkill to use the SOTA models. People want SOTA because it's shiny, but there are a lot of tasks where it's cheaper and more efficient to use other models.

jiggawatts 3mo ago
> but there are a lot of tasks where it's cheaper and more efficient to use other models. Sure… but which ones? How can you know ahead of time? I just did a “simple” upgrade project where both me and the AI kept tripping over dead code, subtle typos, and difficult-to-trace live versus dead code. Many times I used “Medium” thinking I got bitten, but not every time, and I couldn’t predict when. So “Extra high” it was, for the entire project. Far fewer nasty surprises!
- PeterStuer 3mo ago
  In my experience: anything of open-ended complexity (software development, research, product design, ...) benefits from wathever the frontier can offer. 95% of Line of Business automation and workflows can be handled by even a reasonably small open weights generalist model flanked by a few even smaller specialized models. Yes, designing such a setup takes more knowledge and work dan just chucking it all over the api with prompts. But that is how I can run a system here for <$30/month vs >$1.000 month. As an added bonus, no model server can shut me down at the drop of a hat.
- Sammi 3mo ago
  Exactly. I simply don't have the time to deal with non-SOTA model output.
- jfim 3mo ago
  For programmatic usage oftentimes SOTA isn't useful. For example, I have software that summarizes articles and classifies links on webpages to build a synthetic RSS feed, both of which use LLMs, neither of which need a SOTA model. I'll probably use LLMs to bootstrap a dataset of native ads in articles, and there again, I don't really need a SOTA model. If it's for more open ended tasks like writing code though, I agree that at this point SOTA models make more sense to use.
- meetingthrower 3mo ago
  Right. You hire the developer when you want a developer. But if I am building simple agentic workflows -- glorified automations with a small bit of structured "thinking" - I will sure use the cheapest API that can deliver that task at the speed I want. I wonder where the market sizes will shake out for these different types of use cases? I am guessing right now 1 is bigger than 2 but not for long (by token volume)?
- hk__2 3mo ago
  > Sure… but which ones? How can you know ahead of time? Experience, mostly, and bit of trial-and-error. For code I use SOTA, but for e.g. translations I use Gemini 3.5 flash, for some other use-cases I use Gemma 4.

by hk__2 3mo ago

No you don't; it's often overkill to use the SOTA models. People want SOTA because it's shiny, but there are a lot of tasks where it's cheaper and more efficient to use other models.

jiggawatts 3mo ago

> but there are a lot of tasks where it's cheaper and more efficient to use other models. Sure… but which ones? How can you know ahead of time? I just did a “simple” upgrade project where both me and the AI kept tripping over dead code, subtle typos, and difficult-to-trace live versus dead code. Many times I used “Medium” thinking I got bitten, but not every time, and I couldn’t predict when. So “Extra high” it was, for the entire project. Far fewer nasty surprises!

PeterStuer 3mo ago
In my experience: anything of open-ended complexity (software development, research, product design, ...) benefits from wathever the frontier can offer. 95% of Line of Business automation and workflows can be handled by even a reasonably small open weights generalist model flanked by a few even smaller specialized models. Yes, designing such a setup takes more knowledge and work dan just chucking it all over the api with prompts. But that is how I can run a system here for <$30/month vs >$1.000 month. As an added bonus, no model server can shut me down at the drop of a hat.
Sammi 3mo ago
Exactly. I simply don't have the time to deal with non-SOTA model output.
jfim 3mo ago
For programmatic usage oftentimes SOTA isn't useful. For example, I have software that summarizes articles and classifies links on webpages to build a synthetic RSS feed, both of which use LLMs, neither of which need a SOTA model. I'll probably use LLMs to bootstrap a dataset of native ads in articles, and there again, I don't really need a SOTA model. If it's for more open ended tasks like writing code though, I agree that at this point SOTA models make more sense to use.
meetingthrower 3mo ago
Right. You hire the developer when you want a developer. But if I am building simple agentic workflows -- glorified automations with a small bit of structured "thinking" - I will sure use the cheapest API that can deliver that task at the speed I want. I wonder where the market sizes will shake out for these different types of use cases? I am guessing right now 1 is bigger than 2 but not for long (by token volume)?
hk__2 3mo ago
> Sure… but which ones? How can you know ahead of time? Experience, mostly, and bit of trial-and-error. For code I use SOTA, but for e.g. translations I use Gemini 3.5 flash, for some other use-cases I use Gemma 4.