3 ms·

They cheated again: https://news.ycombinator.com/item?id=48847019 https://news.ycombinator.com/item?id=48847019

• meric_ 3mo ago
  No they didn't. Please read: https://www.anthropic.com/engineering/infrastructure-noise https://www.anthropic.com/engineering/infrastructure-noise
  • solarkraft 3mo ago
    Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks: > The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. > An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret. So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.