All physical systems have "intrinsic properties" and "measure properties". Eg., coffee has some internal kinetic energy in the motion of its molecules, and it

All physical systems have "intrinsic properties" and "measure properties". Eg., coffee has some internal kinetic energy in the motion of its molecules, and it has the disposition to cause a thermometer to rise its mercury to a certain height. There's always an open question in these cases: is the height of the mercury a "reliable stand-in" for the temperature of the system? In many case: NO. If you read-off the height too quickly, you'll report the wrong temperature. No system's intrinsic properties is, litearlly, just its measure properties. We are not literally our behaviours. An LLM is not literally its input/output tokens. The question arises: what is the actual intrinsic property which gives rise to the measured properties? it's very easy to see why people believe that the LLM case is parallel to the human case, becuse in ordinary circumstances, our linguistic behvaviours are "reliable measures" of our mental states. So we apply the same perception to LLMs: so to must they generate outputs in the way we do, they must "Reason". However, there are many much more plausible explanations of how LLMs work that do not resort to giving them mental capacities. And so what's left to those in possession of such better explanations is to try to explain to others why you cannot just put a thermometer into a xbox cd drive and think you're measuring how hot the player is.