3 ms·

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less

by andai 7mo ago

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less on the reasoning side as well. Here is a comparison for 4.5, 4.6 and 4.7 (Output Tokens section): https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclaude-opus-4-6-adaptive%2Cclaude-opus-4-5-thinking https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... 4.7 comes out slightly cheaper than 4.6. But 4.5 is about half the cost: https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclaude-opus-4-6-adaptive%2Cclaude-opus-4-5-thinking#cost https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... Notably the cost of reasoning has been cut almost in half from 4.6 to 4.7. I'm not sure what that looks like for most people's workloads, i.e. what the cost breakdown looks like for Claude Code. I expect it's heavy on both input and reasoning, so I don't know how that balances out, now that input is more expensive and reasoning is cheaper. On reasoning-heavy tasks, it might be cheaper. On tasks which don't require much reasoning, it's probably more expensive. (But for those, I would use Codex anyway ;)

matheusmoreira 7mo ago
It thinks less and produces less output tokens because it has forced adaptive thinking that even API users can't disable. Same adaptive thinking that was causing quality issues in Opus 4.6 not even two weeks ago. The one bcherny recommended that people disable because it'd sometimes allocate zero thinking tokens to the model. https://news.ycombinator.com/item?id=47668520 https://news.ycombinator.com/item?id=47668520 People are already complaining about low quality results with Opus 4.7. I'm also spotting it making really basic mistakes. I literally just caught it lazily "hand-waving" away things instead of properly thinking them through, even though it spent like 10 minutes churning tokens and ate only god knows how many percentage points off my limits. > What's the difference between this and option 1.(a) presented before? > Honestly? Barely any. Option M is option 1.(a) with the lifecycle actually worked out instead of hand-waved. > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. > Fair call. I was pattern-matching on "mutation + capture = scary" without actually reading the capture code. Let me do the work properly. > You were right to push back. I was wrong. Let me actually trace it properly this time. > My concern from the first pass was right. The second pass was me talking myself out of it with a bad trace. It's just a constant stream of self-corrections and doubts. Opus simply cannot be trusted when adaptive thinking is enabled. Can provide session feedback IDs if needed.
- rectang 7mo ago
  Are the benchmarks being used to measure these models biased towards completing huge and highly complex tasks, rather than ensuring correctness for less complex tasks? It seems like they're working hard to prioritize wrapping their arms around huge contexts, as opposed to handling small tasks with precision. I prefer to limit the context and the scope of the task and focus on trying to get everything right in incremental steps.
  - matheusmoreira 7mo ago
    I don't think there's a bias here. I'd say my task is of somewhat high complexity. I'm using Claude to assist me in implementing exceptions in my programming language. It's a SICP chapter 5.4 level task. There are quite a few moving parts in this thing. Opus 4.6 once went around in circles for half an hour trying to trace my interpreter's evaluator. As a human, it's not an easy task for me to do either. I think the problem just comes down to adaptive thinking allowing the model to choose how much effort it spends on things, a power which it promptly abuses to be as lazy as possible. CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 significantly improved Opus 4.6's behavior and the quality of its results. But then what do they do when they release 4.7? https://code.claude.com/docs/en/model-config https://code.claude.com/docs/en/model-config > Opus 4.7 always uses adaptive reasoning. > The fixed thinking budget mode and CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING do not apply to it.
    - bobkb 7mo ago
      ‘effort high/max’ seems to be working though
      - matheusmoreira 7mo ago
        The problem I described occurred on Claude Code, Opus 4.7/1M, max effort, patched system prompts with all "don't think for simple stuff" instructions removed as well as CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 even though Opus 4.7 ignores it.
- 7mo ago
QuantumGood 7mo ago
Some have defined "fair" as tests of the same model at different times, as the behavior and token usage of a model changes despite the version number remaining the same. So testing model numbers at different times matters, unfortunately, and that means recent tests might not be what you would want to compare to future tests.
scrollop 7mo ago
That is not what anthropic says- "Opus 4.7 thinks more at higher effort levels, particularly on later turns in agentic settings. This improves its reliability on hard problems, but it does mean it produces more output tokens. " https://www.anthropic.com/news/claude-opus-4-7 https://www.anthropic.com/news/claude-opus-4-7
- andai 7mo ago
  That's a good point. AA's Cost Efficiency section says the opposite: you can hover to see the breakdown between input, reasoning and output tokens. I'm not sure where that discrepancy comes from (is Anthropic using different benchmarks?). There's a few different theories but all we have now are synthetic benchmarks, anecdotes and speculation. (Benchmarks are misleading, I think our best bet now is for individuals to run real world tests, giving the same task to each model, and compare the quality, cost and time.) The input cost inflation however is real, and dramatic. I would have expected them to lower input costs proportionally, because otherwise you're getting less intelligence per dollar even with the smarter model. Think that would be the smartest thing for them to do, at least PR wise. And maybe a bit of free usage as an apology :)
irthomasthomas 7mo ago
The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchmarks are a rebuttal to real usage data.
- andai 7mo ago
  The cost of the input tokens, not the reasoning or output. Agree though that benchmarks aren't very helpful w.r.t. estimating real world performance or costs. What we'd need are people giving the same real world tasks to 4.6 and 4.7 and measuring time, quality and costs.
  - irthomasthomas 7mo ago
    Thanks, that wasn't clear because it mentioned conversations, but it is only measuring the input tokens. So its just measuring the difference in the tokenizer.
HarHarVeryFunny 7mo ago
> For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6 Does it? Anthropic's own announcement says that for the same "effort level" 4.7 does more thinking (i.e uses more output tokens) than 4.6, and they've also increased the default effort level from 4.6 high to 4.7 xhigh. I'm not sure what dominates the cost for a typical mix of agentic coding tasks - input tokens or output ones, but if you are working on an existing project rather than a brand new one, then file input has to be a significant factor and preliminary testing says that the new tokenizer is typically generating 40% or so more tokens for the exact same input. I really have to wonder how much of 4.7's increase in benchmark scores over 4.6 is because the model is actually better trained for these cases, or just because it is using more tokens - more compute and thinking steps - to generate the output. It has to be a mix of the two.

by andai 7mo ago

matheusmoreira 7mo ago

It thinks less and produces less output tokens because it has forced adaptive thinking that even API users can't disable. Same adaptive thinking that was causing quality issues in Opus 4.6 not even two weeks ago. The one bcherny recommended that people disable because it'd sometimes allocate zero thinking tokens to the model. https://news.ycombinator.com/item?id=47668520 https://news.ycombinator.com/item?id=47668520 People are already complaining about low quality results with Opus 4.7. I'm also spotting it making really basic mistakes. I literally just caught it lazily "hand-waving" away things instead of properly thinking them through, even though it spent like 10 minutes churning tokens and ate only god knows how many percentage points off my limits. > What's the difference between this and option 1.(a) presented before? > Honestly? Barely any. Option M is option 1.(a) with the lifecycle actually worked out instead of hand-waved. > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. > Fair call. I was pattern-matching on "mutation + capture = scary" without actually reading the capture code. Let me do the work properly. > You were right to push back. I was wrong. Let me actually trace it properly this time. > My concern from the first pass was right. The second pass was me talking myself out of it with a bad trace. It's just a constant stream of self-corrections and doubts. Opus simply cannot be trusted when adaptive thinking is enabled. Can provide session feedback IDs if needed.

rectang 7mo ago
Are the benchmarks being used to measure these models biased towards completing huge and highly complex tasks, rather than ensuring correctness for less complex tasks? It seems like they're working hard to prioritize wrapping their arms around huge contexts, as opposed to handling small tasks with precision. I prefer to limit the context and the scope of the task and focus on trying to get everything right in incremental steps.
- matheusmoreira 7mo ago
  I don't think there's a bias here. I'd say my task is of somewhat high complexity. I'm using Claude to assist me in implementing exceptions in my programming language. It's a SICP chapter 5.4 level task. There are quite a few moving parts in this thing. Opus 4.6 once went around in circles for half an hour trying to trace my interpreter's evaluator. As a human, it's not an easy task for me to do either. I think the problem just comes down to adaptive thinking allowing the model to choose how much effort it spends on things, a power which it promptly abuses to be as lazy as possible. CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 significantly improved Opus 4.6's behavior and the quality of its results. But then what do they do when they release 4.7? https://code.claude.com/docs/en/model-config https://code.claude.com/docs/en/model-config > Opus 4.7 always uses adaptive reasoning. > The fixed thinking budget mode and CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING do not apply to it.
  - bobkb 7mo ago
    ‘effort high/max’ seems to be working though
    - matheusmoreira 7mo ago
      The problem I described occurred on Claude Code, Opus 4.7/1M, max effort, patched system prompts with all "don't think for simple stuff" instructions removed as well as CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 even though Opus 4.7 ignores it.
7mo ago

QuantumGood 7mo ago

Some have defined "fair" as tests of the same model at different times, as the behavior and token usage of a model changes despite the version number remaining the same. So testing model numbers at different times matters, unfortunately, and that means recent tests might not be what you would want to compare to future tests.

scrollop 7mo ago

That is not what anthropic says- "Opus 4.7 thinks more at higher effort levels, particularly on later turns in agentic settings. This improves its reliability on hard problems, but it does mean it produces more output tokens. " https://www.anthropic.com/news/claude-opus-4-7 https://www.anthropic.com/news/claude-opus-4-7

andai 7mo ago
That's a good point. AA's Cost Efficiency section says the opposite: you can hover to see the breakdown between input, reasoning and output tokens. I'm not sure where that discrepancy comes from (is Anthropic using different benchmarks?). There's a few different theories but all we have now are synthetic benchmarks, anecdotes and speculation. (Benchmarks are misleading, I think our best bet now is for individuals to run real world tests, giving the same task to each model, and compare the quality, cost and time.) The input cost inflation however is real, and dramatic. I would have expected them to lower input costs proportionally, because otherwise you're getting less intelligence per dollar even with the smarter model. Think that would be the smartest thing for them to do, at least PR wise. And maybe a bit of free usage as an apology :)

irthomasthomas 7mo ago

The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchmarks are a rebuttal to real usage data.

andai 7mo ago
The cost of the input tokens, not the reasoning or output. Agree though that benchmarks aren't very helpful w.r.t. estimating real world performance or costs. What we'd need are people giving the same real world tasks to 4.6 and 4.7 and measuring time, quality and costs.
- irthomasthomas 7mo ago
  Thanks, that wasn't clear because it mentioned conversations, but it is only measuring the input tokens. So its just measuring the difference in the tokenizer.

HarHarVeryFunny 7mo ago

> For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6 Does it? Anthropic's own announcement says that for the same "effort level" 4.7 does more thinking (i.e uses more output tokens) than 4.6, and they've also increased the default effort level from 4.6 high to 4.7 xhigh. I'm not sure what dominates the cost for a typical mix of agentic coding tasks - input tokens or output ones, but if you are working on an existing project rather than a brand new one, then file input has to be a significant factor and preliminary testing says that the new tokenizer is typically generating 40% or so more tokens for the exact same input. I really have to wonder how much of 4.7's increase in benchmark scores over 4.6 is because the model is actually better trained for these cases, or just because it is using more tokens - more compute and thinking steps - to generate the output. It has to be a mix of the two.