The question of whether language models can "reason" ultimately comes back to the chinese room argument. There aren't any universally accepted resolutions to th

The question of whether language models can "reason" ultimately comes back to the chinese room argument. There aren't any universally accepted resolutions to that argument either, nor whether those limitations (if they exist) are fundamental. However, the current incarnation of LLMs like ChatGPT fail at straightforward reasoning around things like categorizing sailfish and mathematical proofs. I'm incredibly impressed at how powerful the simplistic methods they use are, but the deficiencies make it clear that it's not abstractly reasoning from definitions in the same way humans are able to.