13 ms·
> how do you argue that these models are not able to reason? I don't make this argument. Benchmarks like CLUTRR[1] show how poorly LLMs do in reasoning. [1] h
by msoad 2y ago
> how do you argue that these models are not able to reason?
I don't make this argument. Benchmarks like CLUTRR[1] show how poorly LLMs do in reasoning.
[1] https://github.com/facebookresearch/clutrr https://github.com/facebookresearch/clutrr
- Last5Digits 2y agoThere is a difference between poor reasoning and no reasoning. SOTA LLMs correctly answer a significant number of these questions correctly. The likelihood of doing so without reasoning is astronomically small. Reasoning in general is not a binary or global property. You aren't surprised when high-schoolers don't, after having learned how to draw 2D shapes, immediately go on to draw 200D hypercubes.
- wrsh07 2y agoGranting that, the original point was that they're not excited about this particular paper unless (for example) it improves the networks' general reasoning abilities. The problem was never "my llm can't do addition" - it can write python code! The problem is "my llm can't solve hard problems that require reasoning"