6 ms·
>>> I'm willing to bet this is within human baselines but either way, researchers really need to start including human baselines in these kinds of papers. They
by unwise-exe 2y ago
>>> I'm willing to bet this is within human baselines but either way, researchers really need to start including human baselines in these kinds of papers.
They should indeed, but is that "I've never seen this before" human baseline, or with prior exposure ("what do you mean I got that wro... oh I see what you did there") or explicit instruction?
- famouswaffles 2y agoI mean baselines to match how the LLMs are being tested in whatever paper (as best as possible). In this case, average scores on the unaltered benchmark and then average scores on each modified benchmark to indicate how much on average human performance drops introducing these details. A "hey, you got that wrong. check again" is fine if the LLMs in the paper are also being prompted that way.