Part of the confusion stems from implicitly pulling in the concept of consciousness into the definition of intelligence. There is an interiority to our thought

Part of the confusion stems from implicitly pulling in the concept of consciousness into the definition of intelligence. There is an interiority to our thought life. At least I know there is for myself, because I know what it's like to experience the world as me. I assume that other humans have this same kind of interiority as me, because they are humans like me. And then animals to greater or lesser extent based on how similarly they behave or sense the world around them to humans. But if there is an "interiority" for LLMs, it must be very very different to humans. The reasoning of an LLM springs into existence for every prompt, then goes away entirely for the next prompt, starting over again from scratch. Yes this is an over simplification. The LLM has been trained with all kinds of knowledge about the world that persists between invocations. But if the floating point numbers are just sitting there on a disk or other storage medium, it doesn't seem possible that it could be experiencing anything until called into use again. And the strangeness of the LLM having a completely transformed personality and biases based solely on a few sentences in a prompt. "You are a character in the Lord of the Rings..." I think this is the sense in which many people argue that an LLM is not "intelligent". It's really an argument that an LLM does not experience the world anything like the way a human being does.