4 ms·

There Are No New Ideas in AI Only New Datasets

3 points by sebg 2y ago

Ukv 2y ago
> In one particularly salient example, when language models were evaluated on the latest math olympiad exam, they scored only 5%, indicating that recent announcements may have been overblown DeepSeek-R1 and Grok-3 scored 5%, but Gemini-2.5-Pro scored 25%. For comparison, the median score of human olympiad competitors last year was 31%[0] (not sure if this year's results are available yet), and median score of humans in general is almost certainly 0% - math olympiad is very difficult. [0]: https://web.evanchen.cc/exams/posted-usamo-statistics.pdf https://web.evanchen.cc/exams/posted-usamo-statistics.pdf