3 ms·
what the hell is that score/cost curve lol
by yusufozkan 3mo ago
what the hell is that score/cost curve lol
- minimaxir 3mo agoDeepSeek v4 Flash recently had a similar "more reasoning is cheaper" curve. It's a fun counterintuition.
- Phemist 3mo agoWhat is the intuition. Higher quality turns due to more reasoning results in significantly fewer turns taken?
- minimaxir 3mo agoYes, in theory.
- tedsanders 3mo agoYep. In particular, ARC-AGI-3 is a series of games where if you fail, you keep trying again (until eventually hitting a timeout). So the sooner you succeed, the sooner you stop spending tokens retrying. If it was a benchmark where everyone got one attempt with no retries, you wouldn't see it bend backward.
- Frost1x 3mo agoIt’s not that different than a lot of real world economies. Often paying for someone or something with better quality can reduce total costs. You have less failures, less mistakes, so on, so while the expertise or quality of the product is higher than cheaper solutions, they can be more reliable and over time ultimately cheaper. The question I have is how far back that curve can go without relying on economies of scale to just drag all the points back to the left. And without overfitting a specific metric that I don’t need (like this test).