99.9% of the time, "secret" in a headline is cheap counter-factual clickbait. Here it is both literally true, and critical to the usefulness of the "benchmark"

99.9% of the time, "secret" in a headline is cheap counter-factual clickbait. Here it is both literally true, and critical to the usefulness of the "benchmark". Which is a math test. A math test which the AI's do not get to cheat on, by having seen in their training sets. A math test which the top AI's perform extremely poorly on. Added: For the top AI's, "stumped" means that they could not solve even 2% of the problems. Vs. for the PhD's, "stumped" means that they needed hours or days to solve the problems. Very different outcomes.