So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know wha

So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know what humans do, no human does, as we don't have such an access to our internal representations. You neither know what LLMs do, no human does, as even though we have full access to their internals, we have little to no idea how to interpret said internals. The most interesting thing about LLMs is the degree that they have learned to model the real world while only being exposed to words. Because those words are highly correlated not just with each other, but with an external entity unknown to the LLM, it's very likely that by learning to model that external entity, the LLM's word-predicting power will increase. LLMs are likely to have converged on a world-model simply because it makes the most sense given the available evidence. This is no different from how human scientists create models of things they cannot directly see based on what they can see. And of course, "seeing" is not directly perceiving the external reality either! Sensory data merely correlates with what's really going on, and the brain has learned to model the world – an unknown external entity – based on the correlations that exist in the sensory language.