3 ms·

Yet https://marginlab.ai/trackers/claude-code/ https://marginlab.ai/trackers/claude-code/ says no issue. If you're so convinced the models keep getting worse,

• datadrivenangel 7mo ago
  Came here to post this as well, and it's interesting to see how benchmarks don't always track feelings. Which is one of the things people say in favor of Anthropic Models!
• deleted 7mo ago
  [deleted]
• Majromax 7mo ago
  If I'm reading that page correctly, then the benchmark results don't cover the interesting "mid February" inflection point noted in the article/report. The numbers appear to begin after the quality drop began. Moreover, the daily confidence interval seems to be stupidly wide, with a confidence interval between 42% and 69%? The "Other metrics" graphs extend for a longer period, and those do seem to correlate with the report. Notably, the 'input tokens' (and consequently API cost) roughly halve (from 120M to 60M) between the beginning of February and mid-March, while the number of output tokens remains similar. That's consistent with the report's observation that new!Opus is more eager to edit code and skips reading/research steps.
• _3u10 7mo ago
  Why bother, i just use opencode now. ai is a commodity.
• siva7 7mo ago
  why should we trust this random bench? i'm usually more sympathetic towards the "it's you who is holding it wrong" crowd but given how anthropic deceived customers recently and that i am a heavy power user with strong insights into many of these products i also can attest the pattern from the gh issue.
• entrep 7mo ago
  One could argue that subscription based inference might differ from per-token billed API usage.