3 ms·
It's really not that hard to not build a custom bench setup to game the benchmark instead of just using your product straight out of the box, though.
by ambicapter 1y ago
It's really not that hard to not build a custom bench setup to game the benchmark instead of just using your product straight out of the box, though.
- deleted 1y ago[deleted]
- jasonjmcghee 1y agoRight. Building a custom setup is blatant- that will wildly overfit. But let's say a group uses it as a metric as part of CI and each new idea / feature they create runs against SWE bench. Maybe they have parameterized bits and pieces they adjust, maybe they have multiple candidates datasets for fine tuning, maybe they're choosing between checkpoints. This will also end up overfitting - especially if done habitually. It might be a great metric and result in a more powerful overall model. Or it might not.
- VikingCoder 1y agoRight, other than financial pressure. Which is, of course, immense.