There's three things at play here - First is humans are bad at grokking complex systems - simple rules applied at large scale can create enormously complex sys

There's three things at play here - First is humans are bad at grokking complex systems - simple rules applied at large scale can create enormously complex systems, and we're bad at predicting what those things are going to be. The second is these systems are "self-tuned" - we explicitly designed the basic structure, but the interactions between different parts of the structure are derived, not created, and so we can't say exactly why those parameters are picked. The final one is that they're simulating language, and we're a species constitutionally prone to anthropomorphizing - if something looks like it's acting with an intent we recognize, we assume it is. That's helpful in an evolutionary sense, but it means it's really easy to fool us into thinking something is sentient. We've been doing this to ourselves since Eliza. The LLMs look impressive, but it takes very little time or effort to notice there's nobody home - not least of which is you need to keep feeding the model everything it's said before and if you tap out of the context window, it forgets. It's a very neat parlor trick, but there's nobody home there, at least not how they're currently constructed and run. I think there's something being captured in how the models are created and trained, but at best, the run model does not allow for consciousness, because there's no continuity and no updating of the model over time.