6 ms·

82.7% on Terminal Bench is crazy

• toephu2 5mo ago
  Is it? There are 5 other models near ~80% and it was achieved in March... which in AI-world seems like a century ago. https://www.tbench.ai/leaderboard/terminal-bench/2.0 https://www.tbench.ai/leaderboard/terminal-bench/2.0
  • ejpir 5mo ago
    those are not verified. I've tried forgecode and I cannot believe they didn't do something to influence the benchmarks
    • GodelNumbering 5mo ago
      Yup, they were found to be sneaking the answer key using agents.md https://debugml.github.io/cheating-agents/#sneaking-the-answer-key-forgecode-on-terminal-bench-2 https://debugml.github.io/cheating-agents/#sneaking-the-answ...