6 ms·

It's not that I agree with everything Chomsky says or writes but I can't shake the feeling that he's much more intelligent, eloquent and gentlemanly than any of

It's not that I agree with everything Chomsky says or writes but I can't shake the feeling that he's much more intelligent, eloquent and gentlemanly than any of his critics.

the-grump 4y ago
+1 I have my own reservations about his opinion but would never state them in the combative and sneering tone often employed by his critics. Can't help but feel like they personally have a problem with him, and they're using this piece as a cudgel to "take a swing".
tgv 4y ago
Even though I largely disagree with Chomsky, he is extremely intelligent. That Hinton and Sejnowski behave like sore losers is a bit of a disappointment. The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation".
- epups 4y ago
  I did not see a problem with Hinton's argument here - perhaps you would care to expand? If we are evaluating LLM's purely on outcome alone, it seems only fair to compare them to humans that don't have fully functioning cognitive capabilities. We wouldn't discredit a schizophrenic person that hallucinates, we know that this person can also see reality for what it is, most of the time. I just don't think that finding absurd examples, usually after careful prompting, is a slam dunk against LLM's at all.
- naasking 4y ago
  > The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation". They can, it's just that Chomsky's dismissal of "weight adaptations" as insufficient is not actually justified by any evidence. The Transformer model augmented with memory is Turing complete, so it can in principle learn to compute anything using only weight adaptations.
  - tgv 4y ago
    No, it cannot. Even CFGs aren't learnable.
    - naasking 4y ago
      * On the Turing Completeness of Modern Neural Network Architectures, https://arxiv.org/abs/1901.03429 https://arxiv.org/abs/1901.03429 * On the Computational Power of Transformers and its Implications in Sequence Modeling, https://arxiv.org/abs/2006.09286 https://arxiv.org/abs/2006.09286
      - tgv 4y ago
        A TM cannot learn an arbitrary CFG. It needs constraints. Those constraints must also exist in our brain. At birth. It's like an NN architecture. Our brains are much more complex than an artificial NN, though. So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
        naasking 4y ago
        > A TM cannot learn an arbitrary CFG. It needs constraints. Well sure, because CFGs can have unbounded depth, and all real brains are finite. But they can learn bounded CFGs: https://arxiv.org/abs/2112.09174 https://arxiv.org/abs/2112.09174 > Our brains are much more complex than an artificial NN, though. Yes, brains have a few orders of magnitude more parameters, and lots of evolutionary baggage. It's not clear how many parameters would be needed without that evolutionary baggage. Time will tell. > So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched. I don't know what precisely you mean by "share functionality", but it doesn't seem like a stretch to suggest they are in some sense isomorphic given Turing equivalence. Rule 110 doesn't look like it should share much with Turing machines or the lambda calculus or SKI combinators, but these are all equivalent in a formal sense.
        tgv 4y ago
- MrPatan 4y ago
  Our vocal cords evolved for language, but the brain remained as rasa as a tabula can be.
  - tgv 4y ago
    We are not born tabula rasa. Did you ever see a calf get born? It knows how to get onto its feet in minutes, find its mother and start feeding. That's hard-wired. Hormone secretion? Hard-wired. Reflexes? Hard-wired. Creating memories? Hard wired. Identification of sound direction? Hard-wired. Face recognition? Probably hard-wired too. Our CNS comes with a ton of instincts and basic processes. Why wouldn't that extend to other areas? In contrast, the world is too complicated to learn through exposure. Something has to structure our learning process. Language is (in principle) not learnable, only when you make enough assumptions. Those assumptions must be coded somewhere in your brain and probably genes. Tabula rasa sounds nice, but it is untenable and utterly lacks evidence.
emptysongglass 4y ago
I think people, Chomsky included, are really missing the point or else find a way to place themselves in the milieau of oppositional thinkers to attract fame and credibility. Chomsky, by the way, is also a notorious supporter of "Russia's interests" in its criminal prosecution of its invasion into Ukraine. So those lauding his intelligence and wisdom should already have serious reservations before diving in. But let's just get right to it. People who rely on LLMs to do their work in domains they understand find them useful in the sense of hefting another useful tool as humans have done throughout their existence. That's it. Among serious users of LLMs as a productive tool there's no wistful wonder that they may be speaking to a truly understanding intelligence. So Chomsky and the rest of the critics don't really have a foot to stand on when they say things like > It is at once comic and tragic, as Borges might have noted, that so much money and attention should be concentrated on so little a thing — something so trivial when contrasted with the human mind, which by dint of language, in the words of Wilhelm von Humboldt, can make “infinite use of finite means”, creating ideas and theories with universal reach. Why is it comic and tragic? LLMs are genuinely useful tools and one of my biggest productivity boosters at work since I learned how to make a computer do my bidding. Of course huge amounts of money are being poured in when it's so helpful across a number of domains for getting work done. Really, folks, there's nothing to see here. Make use of these marvelous tools and continue doing what humans have done best, a truly innate human inheritance, to use tools to support our goals.
- jmull 4y ago
  He directly acknowledges the utility of these models and seems to be trying to make a different, much wider point. From the paragraph above the one you quoted: > However useful these programs may be in some narrow domains (they can be helpful in computer programming, for example, or in suggesting rhymes for light verse), we know from the science of linguistics and the philosophy of knowledge that they differ profoundly from how humans reason and use language. I think you and he agree on your main point, but his main point is about something else.
  - emptysongglass 4y ago
    I declare his wider point isn't very useful to the discourse at large. Most everyone acknowledges and agrees on the fact that LLMs are stochastic parrots. That doesn't diminish their utility. Chomsky is trying to make himself sound sagacious in a sea of hype but he isn't adding anything new to the conversation and even the title, "The False Promise of ChatGPT" (emphasis mine) attacks a contingent of believers that simply don't exist or are a minority. ChatGPT isn't promising to be anything other than useful. It fulfills that promise.
throw_pm23 4y ago
Can't it be that you are fooled by his eloquence and gentlemanliness, but the actual substance is weak? Or that you cut him more slack because you want him to be right? Chomsky used some examples in his article of what LLM's would get wrong. It is clear that he hasn't even tried, because chatGPT in fact gets those examples right. Why does he get a free pass for such errors? Sounds like Aristotle when he claimed that women do not have wisdom teeth for reasons (he made up), instead of bothering to check.
- the-grump 4y ago
  An LLM can get a query right one time, wrong another. You might want to check your wife's teeth :)
optimalsolver 4y ago
I don’t know about gentlemanly or eloquent, but I seriously doubt Chomsky is smarter than Geoff Hinton or Scott Aaronson.
- deleted 4y ago
  [deleted]
- throw_pm23 4y ago
  I think Chomsky is totally wrong on the current issue (and much else), but he did foundational work in linguistics and TCS half a century ago. What did Scott Aaronson contribute of comparable impact?
  - optimalsolver 4y ago
    Aaronson got the ACM prize for his work in quantum computing. Work which will ultimately be more valuable than Chomsky’s soon to discredited theories.
    - throw_pm23 4y ago
      Sorry, but this is misinformed. Chomsky's hierarchy of formal languages will never be discredited, regardless of his silly views on AI or whatever. Quantum computing is "big if true" category, may turn out to be important or irrelevant or a dead end altogether. Some signs point to the latter already.
      - optimalsolver 4y ago
        All I know is if Hinton and Aaronson are calling bullshit, it’s almost certainly bullshit. And the fact that OP thinks Chomsky is smarter than these two brilliant minds is completely laughable.

It's not that I agree with everything Chomsky says or writes but I can't shake the feeling that he's much more intelligent, eloquent and gentlemanly than any of his critics.

the-grump 4y ago

+1 I have my own reservations about his opinion but would never state them in the combative and sneering tone often employed by his critics. Can't help but feel like they personally have a problem with him, and they're using this piece as a cudgel to "take a swing".

tgv 4y ago

Even though I largely disagree with Chomsky, he is extremely intelligent. That Hinton and Sejnowski behave like sore losers is a bit of a disappointment. The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation".

epups 4y ago
I did not see a problem with Hinton's argument here - perhaps you would care to expand? If we are evaluating LLM's purely on outcome alone, it seems only fair to compare them to humans that don't have fully functioning cognitive capabilities. We wouldn't discredit a schizophrenic person that hallucinates, we know that this person can also see reality for what it is, most of the time. I just don't think that finding absurd examples, usually after careful prompting, is a slam dunk against LLM's at all.
naasking 4y ago
> The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation". They can, it's just that Chomsky's dismissal of "weight adaptations" as insufficient is not actually justified by any evidence. The Transformer model augmented with memory is Turing complete, so it can in principle learn to compute anything using only weight adaptations.
- tgv 4y ago
  No, it cannot. Even CFGs aren't learnable.
  - naasking 4y ago
    * On the Turing Completeness of Modern Neural Network Architectures, https://arxiv.org/abs/1901.03429 https://arxiv.org/abs/1901.03429 * On the Computational Power of Transformers and its Implications in Sequence Modeling, https://arxiv.org/abs/2006.09286 https://arxiv.org/abs/2006.09286
    - tgv 4y ago
      A TM cannot learn an arbitrary CFG. It needs constraints. Those constraints must also exist in our brain. At birth. It's like an NN architecture. Our brains are much more complex than an artificial NN, though. So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
      - naasking 4y ago
        > A TM cannot learn an arbitrary CFG. It needs constraints. Well sure, because CFGs can have unbounded depth, and all real brains are finite. But they can learn bounded CFGs: https://arxiv.org/abs/2112.09174 https://arxiv.org/abs/2112.09174 > Our brains are much more complex than an artificial NN, though. Yes, brains have a few orders of magnitude more parameters, and lots of evolutionary baggage. It's not clear how many parameters would be needed without that evolutionary baggage. Time will tell. > So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched. I don't know what precisely you mean by "share functionality", but it doesn't seem like a stretch to suggest they are in some sense isomorphic given Turing equivalence. Rule 110 doesn't look like it should share much with Turing machines or the lambda calculus or SKI combinators, but these are all equivalent in a formal sense.
        tgv 4y ago
MrPatan 4y ago
Our vocal cords evolved for language, but the brain remained as rasa as a tabula can be.
- tgv 4y ago
  We are not born tabula rasa. Did you ever see a calf get born? It knows how to get onto its feet in minutes, find its mother and start feeding. That's hard-wired. Hormone secretion? Hard-wired. Reflexes? Hard-wired. Creating memories? Hard wired. Identification of sound direction? Hard-wired. Face recognition? Probably hard-wired too. Our CNS comes with a ton of instincts and basic processes. Why wouldn't that extend to other areas? In contrast, the world is too complicated to learn through exposure. Something has to structure our learning process. Language is (in principle) not learnable, only when you make enough assumptions. Those assumptions must be coded somewhere in your brain and probably genes. Tabula rasa sounds nice, but it is untenable and utterly lacks evidence.

emptysongglass 4y ago

I think people, Chomsky included, are really missing the point or else find a way to place themselves in the milieau of oppositional thinkers to attract fame and credibility. Chomsky, by the way, is also a notorious supporter of "Russia's interests" in its criminal prosecution of its invasion into Ukraine. So those lauding his intelligence and wisdom should already have serious reservations before diving in. But let's just get right to it. People who rely on LLMs to do their work in domains they understand find them useful in the sense of hefting another useful tool as humans have done throughout their existence. That's it. Among serious users of LLMs as a productive tool there's no wistful wonder that they may be speaking to a truly understanding intelligence. So Chomsky and the rest of the critics don't really have a foot to stand on when they say things like > It is at once comic and tragic, as Borges might have noted, that so much money and attention should be concentrated on so little a thing — something so trivial when contrasted with the human mind, which by dint of language, in the words of Wilhelm von Humboldt, can make “infinite use of finite means”, creating ideas and theories with universal reach. Why is it comic and tragic? LLMs are genuinely useful tools and one of my biggest productivity boosters at work since I learned how to make a computer do my bidding. Of course huge amounts of money are being poured in when it's so helpful across a number of domains for getting work done. Really, folks, there's nothing to see here. Make use of these marvelous tools and continue doing what humans have done best, a truly innate human inheritance, to use tools to support our goals.

jmull 4y ago
He directly acknowledges the utility of these models and seems to be trying to make a different, much wider point. From the paragraph above the one you quoted: > However useful these programs may be in some narrow domains (they can be helpful in computer programming, for example, or in suggesting rhymes for light verse), we know from the science of linguistics and the philosophy of knowledge that they differ profoundly from how humans reason and use language. I think you and he agree on your main point, but his main point is about something else.
- emptysongglass 4y ago
  I declare his wider point isn't very useful to the discourse at large. Most everyone acknowledges and agrees on the fact that LLMs are stochastic parrots. That doesn't diminish their utility. Chomsky is trying to make himself sound sagacious in a sea of hype but he isn't adding anything new to the conversation and even the title, "The False Promise of ChatGPT" (emphasis mine) attacks a contingent of believers that simply don't exist or are a minority. ChatGPT isn't promising to be anything other than useful. It fulfills that promise.

throw_pm23 4y ago

Can't it be that you are fooled by his eloquence and gentlemanliness, but the actual substance is weak? Or that you cut him more slack because you want him to be right? Chomsky used some examples in his article of what LLM's would get wrong. It is clear that he hasn't even tried, because chatGPT in fact gets those examples right. Why does he get a free pass for such errors? Sounds like Aristotle when he claimed that women do not have wisdom teeth for reasons (he made up), instead of bothering to check.

the-grump 4y ago
An LLM can get a query right one time, wrong another. You might want to check your wife's teeth :)

optimalsolver 4y ago

I don’t know about gentlemanly or eloquent, but I seriously doubt Chomsky is smarter than Geoff Hinton or Scott Aaronson.

deleted 4y ago
[deleted]
throw_pm23 4y ago
I think Chomsky is totally wrong on the current issue (and much else), but he did foundational work in linguistics and TCS half a century ago. What did Scott Aaronson contribute of comparable impact?
- optimalsolver 4y ago
  Aaronson got the ACM prize for his work in quantum computing. Work which will ultimately be more valuable than Chomsky’s soon to discredited theories.
  - throw_pm23 4y ago
    Sorry, but this is misinformed. Chomsky's hierarchy of formal languages will never be discredited, regardless of his silly views on AI or whatever. Quantum computing is "big if true" category, may turn out to be important or irrelevant or a dead end altogether. Some signs point to the latter already.
    - optimalsolver 4y ago
      All I know is if Hinton and Aaronson are calling bullshit, it’s almost certainly bullshit. And the fact that OP thinks Chomsky is smarter than these two brilliant minds is completely laughable.