8 ms·
Performance of these LLMs on real life tasks feels very much like students last-minute cramming for Asian style exams. The ability to perfectly regurgitate, wh
by rrr_oh_man 2y ago
Performance of these LLMs on real life tasks feels very much like students last-minute cramming for Asian style exams.
The ability to perfectly regurgitate, while no concept of meaning.
- anilakar 2y agoBasically yet another proof that we have managed to perfectly recreate human stupidity :-)
- cscurmudgeon 2y agoGood students are immune to variations that are discussed in the paper. But most academic tests may not differentiate between them and the crammers.
- falcor84 2y ago> Good students are immune to variations I don't believe that. I'd put some good money that if an excellent student is given an exact question from a previous year, they'll do better (faster & more accurate) on it, than when they're given a variation of it.
- n144q 2y agoI don't think you are betting on the same thing the parent comment is talking about. The assumptions aren't the same to begin with.
- Bjartr 2y agoWhat's the difference between benefitting from seeing previous problems and being worse off when not having a previous problem to go from?
- fn-mote 2y agoThe point is that the “good student” will still do well on the variations, not suffer a 30% decrease in grade.
- falcor84 2y agoI'm not following, why would you assume that a good student taking an exam at the edge of their ability not do significantly better if they trained on the exact same questions (with the same solutions), as opposed to ones that are slightly different? I for one have absolutely struggled as a student when faced with questions that seemed similar to ones from previous exams, but actually had a crucial difference, and by looking at the examples on pages 9&10 in the article, I'm pretty sure I would have been likely to be confused too.
- cscurmudgeon 2y agoBecause good students don’t cram. It is not optimal. Understanding the underlying structure is much easier than surface-level cramming.
- hshshshshsh 2y agoLook into JEE Advanced.
- umeshunni 2y agohttps://openreview.net/forum?id=YHWXlESeS8 https://openreview.net/forum?id=YHWXlESeS8 Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40%. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. I'm curious how something like O1 would perform now.
- whimsicalism 2y agoo3 is able to get 25% on never seen before frontiermath problems. sure, the models do better when the answer is directly in their dataset but they’ve already surpassed the average human in novelty on held out problems
- jvanderbot 2y agoThe average human did zero studying on representative problems. LLMs did a lot.
- whimsicalism 2y agoOkay? We are measuring capabilities.
- tzs 2y agoI don't know anything about frontiermath problems, but for Putnam problems (which is what the submitted article is about) the average human that takes the exam is an undergraduate mathematics or science major who has studied prior Putnam problems and other similar problems recently to specifically prepare for the exam...and the most common score is still 0. At top tier schools the most common score will usually be somewhere in the 0 to 10 range (out of a possible 120).
- gldjmp 2y agoYou can get all of the correct numerical answers for the Putnam and still get a zero, because the reasoning is graded very harshly. The scores measured in this paper are not comparable to actual Putnam scores.
- fldskfjdslkfj 2y ago> never seen before frontiermath problems How do you know that?
- whimsicalism 2y agoBecause that is the whole conceit of how frontiermath is constructed