3 ms·

I wrote up a preliminary whitepaper, but haven’t done a statistically sound test yet. I’ll freely admit this is just a competing theory when I’ve only tried a

• skybrian 2y ago
  As previously shown with “thinking step by step”, changing the prompt is a cheap way to get better results. I think the benchmark they introduced will still be useful, though. Hopefully someone will re-run the benchmark with different prompts.
  • zmgsabst 2y ago
    I’m not sure of the utility: Asking the LLM to do “reasoning” as a single intuitive step is fundamentally not assessing its potential to reason — and is unlike what we expect from humans where we insist they “show their work”. (As a math grader, I’ve penalized humans for doing what they’re asking of the LLM.) If we don’t assess human reasoning in the manner they’re using (zero work shown and not establishing clear context), then I don’t see why it would be useful to assess LLM reasoning in that manner. I think GSM-NoOp is a neat idea, but their testing methodology seems like a “gotcha” that wouldn’t be appropriate for human testing either.