One big difference is that LLMs don't do loops, new words are produced in a fixed amount of steps. They can't go "give me a minute", think for a bit in the back

One big difference is that LLMs don't do loops, new words are produced in a fixed amount of steps. They can't go "give me a minute", think for a bit in the background and come back with an answer. That's why you can often get better answers when you force them to go through the individual steps, as that allows them to use the prompt history as working memory. That said, this all seems like a 'fallacy of composition'. Humans are not LLMs, so much should be obvious, but at the same time concluding that they are completely different just feels wrong. The mistakes LLMs make feel very similar to what humans do when they don't have the time to deeply think about a problem and just give you the best guess that pops into their head. Humans will have other systems on top that allow them deeper reasoning, but the language generation really doesn't feel all that different from what LLMs do. That aside, humans interact with the world, they get instant feedback on what of their predictions is right or wrong. LLMs are stuck with just static training data that might simply not be enough to develop higher level reasoning skills.