4 ms·

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-ta

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-task#cost-tabs https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

jjcm 3mo ago
Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k. What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right. Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets? Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.
- qznc 3mo ago
  Like this, i guess: https://entropicthoughts.com/updated-llm-benchmark https://entropicthoughts.com/updated-llm-benchmark Just for coding instead of text adventures.
schaefer 3mo ago
It's still spooky to see exponential scales on the money axis. I do not have exponential funds in my allowance...
- jbs789 3mo ago
  “Allowance” has tipped me off / provided a hint perhaps? If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models. It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time. Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.
  - killingtime74 3mo ago
    Um I get an allowance for AI at work, that's probably what they mean?
    - jbs789 3mo ago
      Really, that’s the term they use? Man, I’d feel like a child if my boss gave me an “allowance”. Wasn’t aware. Thank you.
      - killingtime74 3mo ago
        What does your company call it. We also get an on-call allowance, meal allowance during traveling and mobile phone allowance.
- Der_Einzige 3mo ago
  In terms of marginal utility you sure do.

jjcm 3mo ago

Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k. What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right. Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets? Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.

qznc 3mo ago
Like this, i guess: https://entropicthoughts.com/updated-llm-benchmark https://entropicthoughts.com/updated-llm-benchmark Just for coding instead of text adventures.

schaefer 3mo ago

It's still spooky to see exponential scales on the money axis. I do not have exponential funds in my allowance...

jbs789 3mo ago
“Allowance” has tipped me off / provided a hint perhaps? If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models. It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time. Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.
- killingtime74 3mo ago
  Um I get an allowance for AI at work, that's probably what they mean?
  - jbs789 3mo ago
    Really, that’s the term they use? Man, I’d feel like a child if my boss gave me an “allowance”. Wasn’t aware. Thank you.
    - killingtime74 3mo ago
      What does your company call it. We also get an on-call allowance, meal allowance during traveling and mobile phone allowance.
Der_Einzige 3mo ago
In terms of marginal utility you sure do.