3 ms·
what the hell is that score/cost curve lol
by yusufozkan 1mo ago
what the hell is that score/cost curve lol
- minimaxir 1mo agoDeepSeek v4 Flash recently had a similar "more reasoning is cheaper" curve. It's a fun counterintuition.
- Phemist 1mo agoWhat is the intuition. Higher quality turns due to more reasoning results in significantly fewer turns taken?
- minimaxir 1mo agoYes, in theory.
- tedsanders 1mo agoYep. In particular, ARC-AGI-3 is a series of games where if you fail, you keep trying again (until eventually hitting a timeout). So the sooner you succeed, the sooner you stop spending tokens retrying. If it was a benchmark where everyone got one attempt with no retries, you wouldn't see it bend backward.
- Frost1x 1mo agoIt’s not that different than a lot of real world economies. Often paying for someone or something with better quality can reduce total costs. You have less failures, less mistakes, so on, so while the expertise or quality of the product is higher than cheaper solutions, they can be more reliable and over time ultimately cheaper. The question I have is how far back that curve can go without relying on economies of scale to just drag all the points back to the left. And without overfitting a specific metric that I don’t need (like this test).