4 ms·

Chomsky on the topic: "It simply shows they’re irrelevant. A two-year-old child or two, three-year-old child has basically mastered the essentials of language w

Chomsky on the topic: "It simply shows they’re irrelevant. A two-year-old child or two, three-year-old child has basically mastered the essentials of language with very sparse data. We want to understand language learning, cognition. We want to see how that works. The fact that some program that scans extraordinary amounts of data gets something superficially similar to what a two-year-old child does basically tells us nothing." and "It’s like asking whether submarines swim. You want to call that swimming? Okay, submarine swim. What the programs are doing is, again, scanning huge amounts of data, finding statistical regularities enough so that you can make a fair guess as to what the next word will be in some sequence. Is that thinking? Does submarine swim? It’s the same question."

jongjong 3y ago
I feel like people who play down the significance of AI are lacking a critical piece of understanding about how scalability changes things. Even if it was only as smart as a dog, if you can duplicate it without limit in silicon, that is a highly transformative tech because electricity is cheaper than food. You can solve more problems per unit of currency. Maybe it would be constrained to certain simple problems but still, the AI would completely dominate that problem space and push humans out of it
- lacrimacida 3y ago
  >AI would completely dominate that problem space and push humans out of it This type of AI needs to be trained on human data. If humans are pushed out I don’t see it progressing too far
  - xyzzy123 3y ago
    I feel like AlphaGo is a counter-example - it learned by playing against itself. That said, go is an extremely constrained problem space. It's possible that in science, mathematics, games etc that AI could keep advancing without human input or feedback (and not hit a plateau where more human input is needed) because the universe or whatever axioms you choose provide all the necessary structure you need to keep making "progress". But for cultural artefacts - music / art / literature etc perhaps not. At least in the near term while we provide data, power and compute perhaps the most important thing humans do is set goals and rewards.
  - pixl97 3y ago
    Eh they are already using LLMs in self training. This and with multimodal AI you're not dependent on human generated language but instead "real world" input.
pixl97 3y ago
Otherwise known as "Chomsky being excessively dismissive at recent breakthroughs in AI and language" Now, in some ways this is understandable, Chomsky is interested in language in humans and how we learn with sparse data. But in terms of machine learning and understanding, the machines don't have the same power limitations the human body does. If a machine needs more horsepower it doesn't need to have a million years of evolution to get to that state. If by applying a gigawatt of power I can process all written human works in a reasonable time and I can do that in a shorter amount of time then learning how humans do it with sparse data, then it's well worth the effort.
- AndyNemmity 3y ago
  It's Chomsky being dismissive at LLMs having something to tell us about how humans learn language, which they do not. You're responding in agreement with Chomsky. That by spending a lot of money on Submarines, they can swim better than humans, then it's well worth the effort. He agrees. But he says it tells you nothing about how humans swim.
markisus 3y ago
Chomsky gets it wrong here. A fairer analogy can be made by first supposing that submarines do not exist yet, and people are intensely interested if one could actually be built. Add in the situation that many people are claiming that such a swimming machine is inherently impossible because any machine would lack the swim-enabling "stuff" that fish have. Then one day submarines are built. It passes all tests of swimming ability that naysayers said were impossible for a machine to pass. And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it.
- notatoad 3y ago
  i think "Chomsky gets it right by accident" is maybe better than "Chomsky gets it wrong". "do LLMs think" and "do submarines swim" is kind of the same question - it's a fixation on terminology rather than results. submarines accomplish something useful, and you can either quibble about what terminology to apply to that, or you can use them for the things they're good for and not be super bothered about what it's called. it seems to me that humans have a vested interest in defining thinking as an innately human thing, so it's impossible to build a truly thinking machine, because if you do we'll simply find a new definition of thinking that excludes your machine.
- foobarqux 3y ago
  > And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it. Yes, if you are studying how fish swim it's completely irrelevant that you can build a machine to swim. Chomsky has been making this point for decades. He has never said you could not statistically approximate a natural process, in fact he has explicitly said that you probably could, it's just that it's irrelevant for scientific understanding.
  - drdaeman 3y ago
    Just curious. And how far/detailed is modern scientific understanding of human cognition? I have an impression that, generally speaking, unlike how we know how an LLM operates, we barely have any clues on how humans form thoughts - or am I wrong here? Just some overall ideas that it's "probably somewhere out there, where fMRI shows the activity" but nothing concrete?
jameshart 3y ago
Chomsky here is dismissive because he mistakenly assumes everyone else shares his goal: "We want to understand language learning, cognition" Other people are happy using these things even if they don't advance our understanding of cognition and language learning. And if this current thread of development somehow results in something that functionally amounts to AGI, without advancing our knowledge of cognition, while that would be sad for Chomsky, it would nonetheless be revolutionary technology that would transform society.
- azinman2 3y ago
  He’s not mistaken, he’s just writing for the advancement of linguistics and cognitive science. If you’re interested in robots that’s different.
  - AndyNemmity 3y ago
    Well said. Maybe I should have made it clearer, but I read the article, so I thought it was obvious in a thread about the article.
  - jameshart 3y ago
    Well, the reason a lot of people were interested in computational linguistics and cognition was probably because they want to make robots, and thought that they might need that. Rodney Brooks certainly is interested in making robots in general. I guess if LLMs mean they might be able to get robots without needing cognitive science and linguistics, that's sad for the people in linguistics and cognitive science who have lost that audience for their work now?
- AndyNemmity 3y ago
  The article said LLMs make Chomsky incorrect about how humans learn language. I posted quotes of his indicating that isn't true. Because it isn't true. It's absurdly obvious it isn't true. Everyone replying is talking about other things instead of the topic. LLMs have added zero to the learning about how humans learn language. They've added other value.
bastawhiz 3y ago
This smells like ego to me. He said "you can't have X without Y, it's impossible" and then we got something that kind of has X without Y. Rather than consider that his ideas may be flawed, he's doubling down. And his new statements will almost certainly be proven false when ML research progresses in another five years and machines can learn at a rate that matches or exceeds human children. I'm not the smartest guy, but I know enough to never bet against future technological progress. Is there any empirical evidence that Chomsky's ideas are correct? Or is he setting himself up for embarrassment?
- foobarqux 3y ago
  Yes there is lots of empirical evidence, children acquire language with very little data, all languages are structured in basically the same hierarchical way, etc. You can try reading what he's written.
  - jemmyw 3y ago
    Humans do take quite a long time to learn though, so maybe the data is sparse but still substantial.
    - foobarqux 3y ago
      The amount of data that humans are exposed to isn't enough to rule out lots of candidate grammars yet humans land on specific types of grammars. Also, infants acquire some elements of language extremely rapidly.
- AndyNemmity 3y ago
  You didn't understand what he said. The topic is on universal grammar as a way humans learn language. He is saying that LLMs tell you nothing about how humans learn language, and he is correct. It tells you as much as a submarine tells you about humans swim. It doesn't.
  - deleted 3y ago
    [deleted]

jongjong 3y ago

I feel like people who play down the significance of AI are lacking a critical piece of understanding about how scalability changes things. Even if it was only as smart as a dog, if you can duplicate it without limit in silicon, that is a highly transformative tech because electricity is cheaper than food. You can solve more problems per unit of currency. Maybe it would be constrained to certain simple problems but still, the AI would completely dominate that problem space and push humans out of it

lacrimacida 3y ago
>AI would completely dominate that problem space and push humans out of it This type of AI needs to be trained on human data. If humans are pushed out I don’t see it progressing too far
- xyzzy123 3y ago
  I feel like AlphaGo is a counter-example - it learned by playing against itself. That said, go is an extremely constrained problem space. It's possible that in science, mathematics, games etc that AI could keep advancing without human input or feedback (and not hit a plateau where more human input is needed) because the universe or whatever axioms you choose provide all the necessary structure you need to keep making "progress". But for cultural artefacts - music / art / literature etc perhaps not. At least in the near term while we provide data, power and compute perhaps the most important thing humans do is set goals and rewards.
- pixl97 3y ago
  Eh they are already using LLMs in self training. This and with multimodal AI you're not dependent on human generated language but instead "real world" input.

pixl97 3y ago

Otherwise known as "Chomsky being excessively dismissive at recent breakthroughs in AI and language" Now, in some ways this is understandable, Chomsky is interested in language in humans and how we learn with sparse data. But in terms of machine learning and understanding, the machines don't have the same power limitations the human body does. If a machine needs more horsepower it doesn't need to have a million years of evolution to get to that state. If by applying a gigawatt of power I can process all written human works in a reasonable time and I can do that in a shorter amount of time then learning how humans do it with sparse data, then it's well worth the effort.

AndyNemmity 3y ago
It's Chomsky being dismissive at LLMs having something to tell us about how humans learn language, which they do not. You're responding in agreement with Chomsky. That by spending a lot of money on Submarines, they can swim better than humans, then it's well worth the effort. He agrees. But he says it tells you nothing about how humans swim.

markisus 3y ago

Chomsky gets it wrong here. A fairer analogy can be made by first supposing that submarines do not exist yet, and people are intensely interested if one could actually be built. Add in the situation that many people are claiming that such a swimming machine is inherently impossible because any machine would lack the swim-enabling "stuff" that fish have. Then one day submarines are built. It passes all tests of swimming ability that naysayers said were impossible for a machine to pass. And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it.

notatoad 3y ago
i think "Chomsky gets it right by accident" is maybe better than "Chomsky gets it wrong". "do LLMs think" and "do submarines swim" is kind of the same question - it's a fixation on terminology rather than results. submarines accomplish something useful, and you can either quibble about what terminology to apply to that, or you can use them for the things they're good for and not be super bothered about what it's called. it seems to me that humans have a vested interest in defining thinking as an innately human thing, so it's impossible to build a truly thinking machine, because if you do we'll simply find a new definition of thinking that excludes your machine.
foobarqux 3y ago
> And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it. Yes, if you are studying how fish swim it's completely irrelevant that you can build a machine to swim. Chomsky has been making this point for decades. He has never said you could not statistically approximate a natural process, in fact he has explicitly said that you probably could, it's just that it's irrelevant for scientific understanding.
- drdaeman 3y ago
  Just curious. And how far/detailed is modern scientific understanding of human cognition? I have an impression that, generally speaking, unlike how we know how an LLM operates, we barely have any clues on how humans form thoughts - or am I wrong here? Just some overall ideas that it's "probably somewhere out there, where fMRI shows the activity" but nothing concrete?

jameshart 3y ago

Chomsky here is dismissive because he mistakenly assumes everyone else shares his goal: "We want to understand language learning, cognition" Other people are happy using these things even if they don't advance our understanding of cognition and language learning. And if this current thread of development somehow results in something that functionally amounts to AGI, without advancing our knowledge of cognition, while that would be sad for Chomsky, it would nonetheless be revolutionary technology that would transform society.

azinman2 3y ago
He’s not mistaken, he’s just writing for the advancement of linguistics and cognitive science. If you’re interested in robots that’s different.
- AndyNemmity 3y ago
  Well said. Maybe I should have made it clearer, but I read the article, so I thought it was obvious in a thread about the article.
- jameshart 3y ago
  Well, the reason a lot of people were interested in computational linguistics and cognition was probably because they want to make robots, and thought that they might need that. Rodney Brooks certainly is interested in making robots in general. I guess if LLMs mean they might be able to get robots without needing cognitive science and linguistics, that's sad for the people in linguistics and cognitive science who have lost that audience for their work now?
AndyNemmity 3y ago
The article said LLMs make Chomsky incorrect about how humans learn language. I posted quotes of his indicating that isn't true. Because it isn't true. It's absurdly obvious it isn't true. Everyone replying is talking about other things instead of the topic. LLMs have added zero to the learning about how humans learn language. They've added other value.

bastawhiz 3y ago

This smells like ego to me. He said "you can't have X without Y, it's impossible" and then we got something that kind of has X without Y. Rather than consider that his ideas may be flawed, he's doubling down. And his new statements will almost certainly be proven false when ML research progresses in another five years and machines can learn at a rate that matches or exceeds human children. I'm not the smartest guy, but I know enough to never bet against future technological progress. Is there any empirical evidence that Chomsky's ideas are correct? Or is he setting himself up for embarrassment?

foobarqux 3y ago
Yes there is lots of empirical evidence, children acquire language with very little data, all languages are structured in basically the same hierarchical way, etc. You can try reading what he's written.
- jemmyw 3y ago
  Humans do take quite a long time to learn though, so maybe the data is sparse but still substantial.
  - foobarqux 3y ago
    The amount of data that humans are exposed to isn't enough to rule out lots of candidate grammars yet humans land on specific types of grammars. Also, infants acquire some elements of language extremely rapidly.
AndyNemmity 3y ago
You didn't understand what he said. The topic is on universal grammar as a way humans learn language. He is saying that LLMs tell you nothing about how humans learn language, and he is correct. It tells you as much as a submarine tells you about humans swim. It doesn't.
- deleted 3y ago
  [deleted]