5 ms·

Textual AIs can do text really well. I would argue that even that they can’t do well at all. Look at the training set for a human. Humans learn speech, reading

Textual AIs can do text really well. I would argue that even that they can’t do well at all. Look at the training set for a human. Humans learn speech, reading, and writing with a training set so small it’d be a rounding error from zero compared to what they feed into LLMs. Chomsky called this the “poverty of the stimulus” in his argument against behaviourism. Yet LLMs are fundamentally the product of Skinner-style reinforcement learning. And it shows. They have succeeded to a wonderful degree at producing a facsimile of language understanding. But possession of language? Absolutely not!

alganet 3y ago
Using AI behavior in an argument about psychology seems like a slippery slope though.
- chongli 3y ago
  Chomsky didn't talk about AI at all in his argument about human language acquisition. Poverty of the stimulus refers to the very limited language exposure children are given when learning to speak. It's an extremely powerful argument in favour of an innate capacity for language, and more specifically an innate knowledge of grammar, in the human brain.
  - alganet 3y ago
    I see now, thanks!
cameronh90 3y ago
Humans have a huge base model trained over billions of years of evolution. It’s impressive how quickly we learn, but it’s arguably comparable to fine tuning an existing model rather than starting from scratch. Conversely, I find it insanely impressive how human-like LLMs have become by doing little more than feeding a vast quantity of reading material into a very basic neural architecture. I wouldn’t have ever believed how successful that approach would be.
- chongli 3y ago
  The human model isn’t just trained in language though, it’s trained on the world. There’s such a huge amount of tacit knowledge we have about how the world works and how other people work. We have models for recognizing faces and reading body language, emotions, and intent. We can tell when a person is looking at us from across the room and tell that apart from someone looking just over our shoulder at someone else. I think most of what we ascribe to intelligence in LLMs is a kind of textual pareidolia. We infer a mind behind the text because our brains are built to recognize minds. That’s how our theory of mind works. But these LLMs know nothing about the world. They might capture some features of the world as patterns that help them better predict what text to generate for us. But that is not the same as knowledge of the world. They have no sensory contact with the world so they are unable to synthesize true knowledge about it. This is why they are so prone to hallucination.
  - jamiek88 3y ago
    > I think most of what we ascribe to intelligence in LLMs is a kind of textual pareidolia This is a great way of expressing a thought I’ve been unable to fully articulate.
  - kordlessagain 3y ago
    So we are model ensembles, probably capable of real-time training of short term memory. LLMs are maybe a type of mind. They may carry and operate on basic thoughts, ideas, books, pages, sentences, words, letters…. They can probably problem solve and write code that runs sometimes. What they know about the world was told to them through stories, so whatever way we see it, they will too, maybe. I don’t have to imagine anything in mind to process thoughts. All I need is an attention loop.
    - chongli 3y ago
      What they know about the world was told to them through stories, so whatever way we see it, they will too, maybe. Knowledge is justified true belief. If you park your car on 5th avenue, you know it is parked there because you witnessed yourself parking it there. You have strong justification for your belief that you parked it on 5th avenue. But if you then go to work and your car is out of sight, your justification fades. If you look out the window and see your car parked there, you reaffirm your belief. However, if you look and your car is not there, you realize you don’t know where your car is. For a period of time you were mistaken in your belief about the whereabouts of your car. During the time when your car was out of sight, you went from knowing to not knowing where your car was, but nothing changed inside your mind. To update your belief, you needed new information about the world. An LLM has no justification for any of the information in its training set. It has no experiences with which to connect words and facts to real world objects and events. Therefore it cannot be said to have knowledge of any of it.

alganet 3y ago

Using AI behavior in an argument about psychology seems like a slippery slope though.

chongli 3y ago
Chomsky didn't talk about AI at all in his argument about human language acquisition. Poverty of the stimulus refers to the very limited language exposure children are given when learning to speak. It's an extremely powerful argument in favour of an innate capacity for language, and more specifically an innate knowledge of grammar, in the human brain.
- alganet 3y ago
  I see now, thanks!

cameronh90 3y ago

Humans have a huge base model trained over billions of years of evolution. It’s impressive how quickly we learn, but it’s arguably comparable to fine tuning an existing model rather than starting from scratch. Conversely, I find it insanely impressive how human-like LLMs have become by doing little more than feeding a vast quantity of reading material into a very basic neural architecture. I wouldn’t have ever believed how successful that approach would be.

chongli 3y ago
The human model isn’t just trained in language though, it’s trained on the world. There’s such a huge amount of tacit knowledge we have about how the world works and how other people work. We have models for recognizing faces and reading body language, emotions, and intent. We can tell when a person is looking at us from across the room and tell that apart from someone looking just over our shoulder at someone else. I think most of what we ascribe to intelligence in LLMs is a kind of textual pareidolia. We infer a mind behind the text because our brains are built to recognize minds. That’s how our theory of mind works. But these LLMs know nothing about the world. They might capture some features of the world as patterns that help them better predict what text to generate for us. But that is not the same as knowledge of the world. They have no sensory contact with the world so they are unable to synthesize true knowledge about it. This is why they are so prone to hallucination.
- jamiek88 3y ago
  > I think most of what we ascribe to intelligence in LLMs is a kind of textual pareidolia This is a great way of expressing a thought I’ve been unable to fully articulate.
- kordlessagain 3y ago
  So we are model ensembles, probably capable of real-time training of short term memory. LLMs are maybe a type of mind. They may carry and operate on basic thoughts, ideas, books, pages, sentences, words, letters…. They can probably problem solve and write code that runs sometimes. What they know about the world was told to them through stories, so whatever way we see it, they will too, maybe. I don’t have to imagine anything in mind to process thoughts. All I need is an attention loop.
  - chongli 3y ago
    What they know about the world was told to them through stories, so whatever way we see it, they will too, maybe. Knowledge is justified true belief. If you park your car on 5th avenue, you know it is parked there because you witnessed yourself parking it there. You have strong justification for your belief that you parked it on 5th avenue. But if you then go to work and your car is out of sight, your justification fades. If you look out the window and see your car parked there, you reaffirm your belief. However, if you look and your car is not there, you realize you don’t know where your car is. For a period of time you were mistaken in your belief about the whereabouts of your car. During the time when your car was out of sight, you went from knowing to not knowing where your car was, but nothing changed inside your mind. To update your belief, you needed new information about the world. An LLM has no justification for any of the information in its training set. It has no experiences with which to connect words and facts to real world objects and events. Therefore it cannot be said to have knowledge of any of it.