5 ms·
The average human did zero studying on representative problems. LLMs did a lot.
by jvanderbot 2y ago
The average human did zero studying on representative problems. LLMs did a lot.
- whimsicalism 2y agoOkay? We are measuring capabilities.
- tzs 2y agoI don't know anything about frontiermath problems, but for Putnam problems (which is what the submitted article is about) the average human that takes the exam is an undergraduate mathematics or science major who has studied prior Putnam problems and other similar problems recently to specifically prepare for the exam...and the most common score is still 0. At top tier schools the most common score will usually be somewhere in the 0 to 10 range (out of a possible 120).
- gldjmp 2y agoYou can get all of the correct numerical answers for the Putnam and still get a zero, because the reasoning is graded very harshly. The scores measured in this paper are not comparable to actual Putnam scores.