3 ms·
New secret math benchmark stumps AI models and PhDs alike
- deleted 2y ago[deleted]
- bell-cot 2y ago99.9% of the time, "secret" in a headline is cheap counter-factual clickbait. Here it is both literally true, and critical to the usefulness of the "benchmark". Which is a math test. A math test which the AI's do not get to cheat on, by having seen in their training sets. A math test which the top AI's perform extremely poorly on. Added: For the top AI's, "stumped" means that they could not solve even 2% of the problems. Vs. for the PhD's, "stumped" means that they needed hours or days to solve the problems. Very different outcomes.
- jqpabc123 2y agoThis benchmark is designed to show that current AI lacks reasoning ability and doesn't really "solve" anything. Retrieving a solution that happens to exist in a large database is not the same as "solving" the problem --- though it can appear otherwise to the ill-informed.