3 ms·
Really long-term task benchmark showing significant improvements in very recent models, while also showing really bad regression rates across the board.
by verdverm 8mo ago
Really long-term task benchmark showing significant improvements in very recent models, while also showing really bad regression rates across the board.
- woeirua 8mo agoUh, Opus 4.6 avoids introducing regressions 75% of the time?
- verdverm 8mo agoSo 1/4 times it does not introduce a regression. That's still pretty bad imo. If 1/4 commits introduced regressions, what would your team do? We are talking about regressions, what once worked no longer does, and should be measured in 9s
- notduncansmith 8mo agoYou overestimate many teams I think.