3 ms·
This is silly. Humans will also get fewer right answers if you make the question more complex (requiring additional steps), or if you add irrelevant information
by apsec112 2y ago
This is silly. Humans will also get fewer right answers if you make the question more complex (requiring additional steps), or if you add irrelevant information as a distraction (since on tests, there's usually an assumption that all information given is relevant). As for changing names and numbers, the large effects they saw were all on small (<10B param) open source models; the effects on o1 were tiny and barely distinguishable from noise.
- unwise-exe 2y ago>>> since on tests, there's usually an assumption that all information given is relevant Maybe on grade-school tests, but the professional certification exams I've taken have questions about scenarios where part of the challenge is recognizing which parts of the scenario description aren't relevant. The one I had an in-person class for, the instructor specifically called this out and advised reading the questions first so we'd know which parts of the scenario we didn't have to think about while reading it.