3 ms·
They mentioned in their release page, that the Claude team noticed memorization of the SWE-bench test, so the test is actually in the training data. Here: http
by XCSme 6mo ago
They mentioned in their release page, that the Claude team noticed memorization of the SWE-bench test, so the test is actually in the training data.
Here: https://www.anthropic.com/news/claude-opus-4-7#:~:text=memorization https://www.anthropic.com/news/claude-opus-4-7#:~:text=memor...
- William_BB 6mo agoGood luck arguing with SWE benchmark purists
- sigmoid10 6mo agoAny static benchmark older than 12-18 months is basically worthless, because the content will have spread all over the internet and have found its way into the latest model's training set.