I think the author is wrong. Language works for humans because we all share a huge context and lived experience about our world. Training a model on just the l

I think the author is wrong. Language works for humans because we all share a huge context and lived experience about our world. Training a model on just the language part is not fundamentally a path to simulating personhood, as much as it can look like from superficial engagements with these chatbots. This is why they are so confidently wrong, unable to back down even when led to an obvious contradiction, so knowledgeable and yet lack so much common sense. Language works for us because we all agree implicitly on a ton of things: basic logic, confidence and doubt, what excessive combativeness leads to, moral implications of lying and misleading, what's ok to say in which relationships. There is "knowledge" of this in the weights of GPT3, sure. You can ask it to explain all of the above things and it will. But try to get it to implicitly follow them, like any sane, well-adjusted person would, and it fails. Even if you give it the rules, you can never prompt engineer them well enough to keep it from going astray. I had my own mini-hype-cycle with this thing. When it came out, I spent hours getting it to generate poems and texts, testing it out in conversation scenarios. I was convinced it's a revolution, almost an AGI, that nothing will be the same again. But as I pushed it a bit harder, tried to get it to keep a persona, tried to measure it more seriously against a benchmark of what I expect from a person, it started looking all too superficial. I'm starting to understand the "it's a parlor trick" argument. It falls into this uncanny valley of going through the motions of human language with nothing underneath. It doesn't keep a strong identity and it has a limited context length. Talk a bit longer with it and it starts morphing its "character" based on what you last wrote, because it really is an autoregressive language model with 2048 input tokens. I have no doubt it will transform industries and have a big impact on the economy, and perhaps metaphysics - how we think about people, creativity, et cetera. I do see the author's arguments on that one. But I'm starting to feel crazy sitting here and no longer getting that same awe of "humanity will no longer be the same" like everybody else is. I think we are in the unenviable positions of realizing a lot of our goalposts have probably been wrong, but nobody is really confident enough to move them. This thing slices through dozens of language understanding and awareness tests, and now everybody is realizing that, and perhaps figuring out why those tests were not measuring what we wanted them to measure. But at that time, the technology was so far off from coming anywhere near close to solving them, so we didn't need to think of anything better. Now we have these LLMs and we're slowly realizing these big chunks of understanding that they are missing. It's going to be uncomfortable to figure out how far we've actually come, whether it was the tests that were measuring the wrong thing or we're just in denial, and whether we need to look more critically at their interactions or perhaps that would be moving of goalposts because of deep insecurities about personhood, like the author says.