I tend to agree that LLMs are not thinking in the way that we usually mean it when referring to human thinking. However I think it is dangerous to assert on the

I tend to agree that LLMs are not thinking in the way that we usually mean it when referring to human thinking. However I think it is dangerous to assert on the capabilities of a system based on the structure seemingly imposed by its API. Take for example the instruction set of a CPU. One could argue that a cpu only has N registers or can process only one instruction at a time because the instruction set only contains N registers, and processes instruction linearly. But on any modern application processor, the register renamer allows many more physical registers to be allocated than can be names in the ISA, and instructions are dispatched in parallel to mitigate memory latency and increase throughput. What I mean is that an LLM is not a stochastic parrot because of its API, but rather because it does not outdo a stochastic parrot when tested. This could change though. LLMs could think in full sentences and spoon feed them to us one token at a time, rewording as necessary to provide a few possibilities. They could memorize the letters in each token and count letters correctly despite the limitations imposed by the API.