4 ms·

He’s doing the same to ChatGPT that he did to Skinner in the 1960s. Back then, it put him on the map, and he lives in the illusion that he was right. Now ChatGP

by leobg 3y ago

He’s doing the same to ChatGPT that he did to Skinner in the 1960s. Back then, it put him on the map, and he lives in the illusion that he was right. Now ChatGPT is pretty much Skinner’s work come to life. The BFS book Chomsky critiqued was literally “Verbal Behavior” - a book about how intelligence arises from “dumb” reinforcement learning of words. Obviously Chomsky must now claim that ChatGPT only pretends to be intelligent. Else his entire life’s work is proven wrong.

Al0neStar 3y ago
There is a difference between text and language and so far LLMs have told us nothing about language. LLMs being able to generalize to languages with a much smaller training corpus shows that maybe Chomsky is right about universal grammar.
foldr 3y ago
It’s a significant misunderstanding of Chomsky’s life’s work to think that ChatGPT would prove it wrong. Chomsky’s primary claim is a claim about how language acquisition works in humans. He argues, for example, that certain locality constraints on linguistic dependencies are ‘built in’ and not learned inductively. Thus a human does not ‘learn’ that (i) is ambiguous and (ii) is not: (i) How often did you tell John that he should take out the trash? [how often did you tell, or how often to take it out] (ii) How often did you tell John why he should take out the trash? [only means how often did you tell] Nothing that ChatGPT can do suggests that Chomsky was wrong about this kind of thing. It’s really more of a blow to a certain kind of work in AI that was partly inspired by Chomsky – but not something that he himself ever took much interest in. Now it’s true that Chomsky appears to be in the camp that says ChatGPT doesn’t really understand anything. But the focus of his own work has never been on debunking AI, or making claims about the true nature of understanding, or anything of that ilk.
- Turing_Machine 3y ago
  > He argues, for example, that certain locality constraints on linguistic dependencies are ‘built in’ and not learned inductively. Checking in late here, but one of the pillars of Chomsky's argument is the so-called "poverty of the stimulus" -- basically, that human babies simply don't receive enough training data to acquire language as rapidly and correctly as they demonstrably do. Chomsky therefore concludes that there must be some kind of pre-existing "language module" in the brain to account for this. Now, not everyone accepted this idea even at the time, but surely the argument is much less plausible for an LLM which is likely exposed to more training data than even an adult human.
  - foldr 3y ago
    >Now, not everyone accepted this idea even at the time, but surely the argument is much less plausible for an LLM which is likely exposed to more training data than even an adult human. Yes indeed. Of course this doesn't show that Chomsky was wrong about humans. In any case, I've seen no evidence that current LLMs successfully learn the kinds of constraints I was talking about.
biorach 3y ago
> Now ChatGPT is pretty much Skinner’s work come to life. I'm pretty sure that is based on a misunderstanding of Skinner, ChatGPT, or both
- leobg 3y ago
  Or very lossy compression of what I mean. I have studied both quite extensively. Not 10,000 hours each, but hundreds for sure.
  - goatlover 3y ago
    Did Skinner have anything to say about how the reinforcement works? Because with LLMs you do need the right sort of architecture, and the same with neurons, even though they don't use back propagation. Only humans are known to have language in the full sense, and there has to be some neural reason why that is. Maybe you could make an argument for cetaceans or certain birds, but again they must have the neural architecture for it.
    - foldr 3y ago
      Skinner (and behaviorists in general) did establish various 'laws' of behavioral reinforcement that do tend to hold in simple cases such as pigeons pecking at levers in return for food, etc. etc. Of course these laws had nothing interesting to say about language acquisition. I challenge anyone who thinks otherwise to actually try reading Verbal Behavior. It's an incredibly turgid and uninsightful book.
  - krainboltgreene 3y ago
    The difference between hundreds and 10k hours is roughly 10k hours.

by leobg 3y ago

Al0neStar 3y ago

There is a difference between text and language and so far LLMs have told us nothing about language. LLMs being able to generalize to languages with a much smaller training corpus shows that maybe Chomsky is right about universal grammar.

foldr 3y ago

It’s a significant misunderstanding of Chomsky’s life’s work to think that ChatGPT would prove it wrong. Chomsky’s primary claim is a claim about how language acquisition works in humans. He argues, for example, that certain locality constraints on linguistic dependencies are ‘built in’ and not learned inductively. Thus a human does not ‘learn’ that (i) is ambiguous and (ii) is not: (i) How often did you tell John that he should take out the trash? [how often did you tell, or how often to take it out] (ii) How often did you tell John why he should take out the trash? [only means how often did you tell] Nothing that ChatGPT can do suggests that Chomsky was wrong about this kind of thing. It’s really more of a blow to a certain kind of work in AI that was partly inspired by Chomsky – but not something that he himself ever took much interest in. Now it’s true that Chomsky appears to be in the camp that says ChatGPT doesn’t really understand anything. But the focus of his own work has never been on debunking AI, or making claims about the true nature of understanding, or anything of that ilk.

Turing_Machine 3y ago
> He argues, for example, that certain locality constraints on linguistic dependencies are ‘built in’ and not learned inductively. Checking in late here, but one of the pillars of Chomsky's argument is the so-called "poverty of the stimulus" -- basically, that human babies simply don't receive enough training data to acquire language as rapidly and correctly as they demonstrably do. Chomsky therefore concludes that there must be some kind of pre-existing "language module" in the brain to account for this. Now, not everyone accepted this idea even at the time, but surely the argument is much less plausible for an LLM which is likely exposed to more training data than even an adult human.
- foldr 3y ago
  >Now, not everyone accepted this idea even at the time, but surely the argument is much less plausible for an LLM which is likely exposed to more training data than even an adult human. Yes indeed. Of course this doesn't show that Chomsky was wrong about humans. In any case, I've seen no evidence that current LLMs successfully learn the kinds of constraints I was talking about.

biorach 3y ago

> Now ChatGPT is pretty much Skinner’s work come to life. I'm pretty sure that is based on a misunderstanding of Skinner, ChatGPT, or both

leobg 3y ago
Or very lossy compression of what I mean. I have studied both quite extensively. Not 10,000 hours each, but hundreds for sure.
- goatlover 3y ago
  Did Skinner have anything to say about how the reinforcement works? Because with LLMs you do need the right sort of architecture, and the same with neurons, even though they don't use back propagation. Only humans are known to have language in the full sense, and there has to be some neural reason why that is. Maybe you could make an argument for cetaceans or certain birds, but again they must have the neural architecture for it.
  - foldr 3y ago
    Skinner (and behaviorists in general) did establish various 'laws' of behavioral reinforcement that do tend to hold in simple cases such as pigeons pecking at levers in return for food, etc. etc. Of course these laws had nothing interesting to say about language acquisition. I challenge anyone who thinks otherwise to actually try reading Verbal Behavior. It's an incredibly turgid and uninsightful book.
- krainboltgreene 3y ago
  The difference between hundreds and 10k hours is roughly 10k hours.