4 ms·
I read recently that small variations in the tests cause failures by large margins. If this doesn’t show over fitting in don’t know what would.
by matt3210 2y ago
I read recently that small variations in the tests cause failures by large margins.
If this doesn’t show over fitting in don’t know what would.
- lxgr 2y agoWasn’t that for human tests, i.e. not specifically AI benchmarks? Benchmarks should generally not be game-able by overfitting.
- matt3210 2y agoThe article shows all the tests against human performance. The math one in particular is the one where small variations reduce the success rate significantly. I can’t find the source but it was pasted here in the last 2 weeks.
- CrazyStat 2y agoYou're probably remembering this: https://news.ycombinator.com/item?id=42565606 https://news.ycombinator.com/item?id=42565606
- friend_Fernando 2y agoEventually, all the better AGI tests should have large private evaluation datasets with no possible cheating or feedback loops. We're getting there.