4 ms·
Interesting. I would have thought that the training set (basically the whole internet AIUI) would have included various "teacher's version" exams with enough wo
by unwise-exe 2y ago
Interesting. I would have thought that the training set (basically the whole internet AIUI) would have included various "teacher's version" exams with enough word problems with intentionally-distracting extra information, that the models would be able to ignore that sort of thing.
This sounds like they're inspecting existing models. Maybe a model trained specifically on "word problem" question-answer pairs (as in, the sort of things that show up on tests and always pretend that the sort of complications a domain expert would know about just don't exist) would do better?
- famouswaffles 2y agoThey hid away the results of o1 preview in the Appendix but it does not drop below margin of error numbers on 4/5 of their modified benchmarks. The last one they add "seemingly relevant but ultimately irrelevant information to problems" and it drops to 77%. Now I'm willing to bet this is within human baselines but either way, researchers really need to start including human baselines in these kinds of papers.
- unwise-exe 2y ago>>> I'm willing to bet this is within human baselines but either way, researchers really need to start including human baselines in these kinds of papers. They should indeed, but is that "I've never seen this before" human baseline, or with prior exposure ("what do you mean I got that wro... oh I see what you did there") or explicit instruction?
- famouswaffles 2y agoI mean baselines to match how the LLMs are being tested in whatever paper (as best as possible). In this case, average scores on the unaltered benchmark and then average scores on each modified benchmark to indicate how much on average human performance drops introducing these details. A "hey, you got that wrong. check again" is fine if the LLMs in the paper are also being prompted that way.