This is an interesting acid test: - article is titled "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" - some

This is an interesting acid test: - article is titled "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" - some people having been flogging it as "LLMs cannot reason" - it shows a 6-8 point drop, in test results in the 80s, if you replace the #s in the test set problems with random #s, and run multiple times - If anything, sounds like a huge W to me: very hard to claim they're just memorizing with that small of a drop