I think the difference is more than coherence: it's having complex and rich semantic connections to the rest of the world. I think the coherence and consistency

I think the difference is more than coherence: it's having complex and rich semantic connections to the rest of the world. I think the coherence and consistency you describe is an effect of this. Humans don't just generate text; we interact with the world in all kinds of ways, and those interactions provide us with constant feedback. Furthermore, we can frame hypotheses about how the world works and test them. We can bump up against reality in all kinds of ways that force us to change how we think and how we act. But that constant rich interaction with reality also forces us not to change most of the time--to maintain the coherence and consistency you describe, in order to get along in the world. LLMs have no connections to the rest of the world. All they do is generate text based on patterns in their training data. They don't even have a concept of text being connected to anything else. That's why it's so easy for them to constantly change what they appear to be portraying--there's no anchor to anything else. It's interesting that you call this an Eliza moment, because Eliza's achievement was to fake being a person, by fooling people's heuristics, without having any of the underlying capacities of a real person. LLMs like ChatGPT are indeed doing the same thing. If they're showing us anything, they're showing us how unreliable our intuitive heuristics are as soon as they are confronted with something outside their original domain.