I’m not sure of the utility: Asking the LLM to do “reasoning” as a single intuitive step is fundamentally not assessing its potential to reason — and is unlike

I’m not sure of the utility: Asking the LLM to do “reasoning” as a single intuitive step is fundamentally not assessing its potential to reason — and is unlike what we expect from humans where we insist they “show their work”. (As a math grader, I’ve penalized humans for doing what they’re asking of the LLM.) If we don’t assess human reasoning in the manner they’re using (zero work shown and not establishing clear context), then I don’t see why it would be useful to assess LLM reasoning in that manner. I think GSM-NoOp is a neat idea, but their testing methodology seems like a “gotcha” that wouldn’t be appropriate for human testing either.