4 ms·
I don't see any reason to assume they removed it unless they're very explicit about it. Model publishers have an extremely strong vested interest in beating ben
by jprete 2y ago
I don't see any reason to assume they removed it unless they're very explicit about it. Model publishers have an extremely strong vested interest in beating benchmarks and I expect them to teach to the test if they can get away with it.
- stingraycharles 2y agoAs usual, once a metric becomes a target, it stops being useful.
- franktankbank 2y agoWell, they are doing BigCorpStuff not Science
- whimsicalism 2y agoputnam isn’t an llm benchmark ahhhh none of these companies are reporting putnam scores there’s nothing nefarious about training on putnam problems
- jprete 2y agoAny problem set that can make news is implicitly an LLM benchmark.