3 ms·

I'm regularly in touch with Chomsky. I've interviewed him several times, the latest being this for example with David Harvey: https://www.youtube.com/watch?v=ez

by EGreg 4y ago

I'm regularly in touch with Chomsky. I've interviewed him several times, the latest being this for example with David Harvey: https://www.youtube.com/watch?v=ezf7wxJ7whA https://www.youtube.com/watch?v=ezf7wxJ7whA Earlier regarding Freedom of Speech and Capitalism: https://qbix.com/chomsky https://qbix.com/chomsky Sometimes I send him articles about how chimpanzees or others have "language". Or now they found bumblebees can teach each other. Chomsky famously maintains that only human are born with innate capabilities for language. Anyway, since Chomsky is a linguist and focused on language, it would make sense for him to say that. Of course, computers can develop their own languages (as Facebook's sales bots have done for example years ago: https://nypost.com/2017/08/01/creepy-facebook-bots-talked-to-each-other-in-a-secret-language/ https://nypost.com/2017/08/01/creepy-facebook-bots-talked-to...) I think that, in general, Chomsky is right that when it comes to language (unlike paintings or even photorealistic fakes etc.) the meaning will never be modeled perfectly, any more than, say modeling a Mandelbrot set on all levels of zoom can be done by a machine learning system that trains in the way generative LLMs train. Having said that, I think that logic itself is a "poor man's approximation" to what AI can do, in that it just uses a few parameters. I prefer Steven Wolfram's analysis. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

initramfs 4y ago
I don't think AI can ever learn the many dialects and slang words with quaternary meanings, since statistical methods are quite unreliable on the long tail. I once tried using Google Translate for a word known only in a specific region of a foreign country and got zero results. It's going to be quite complex, esp for non-English languages.
- cma 4y ago
  This seems to show with scale one-shot stuff like hearing a slang word substitution one time can be done (within the other limitations of transformers) https://arxiv.org/abs/2303.03846 https://arxiv.org/abs/2303.03846
calf 4y ago
Logic is just a different level of abstraction, and the current generation of AIs lack it. Think of it as trying to implement an algorithm in machine code versus writing it in C++ or Haskell. Neural networks are analogous to low-level machine code. Chomsky is saying that that's a reductive and insightful and inefficient way of doing things. Human intelligence works on a higher level of abstraction--even though at the same time our brain cells are literally low-level computational abstractions as well. The two levels coexist in humans, whereas in ChatGPT it took great lengths to produce some semblance of that linguistic structure. Chomsky explains his argument differently but his point is that humans have access to a higher level of computational abstraction, and that is the source of efficiency (we don't need to be trained on thousands of cat pictures for this information-theoretic reason).

by EGreg 4y ago

initramfs 4y ago

I don't think AI can ever learn the many dialects and slang words with quaternary meanings, since statistical methods are quite unreliable on the long tail. I once tried using Google Translate for a word known only in a specific region of a foreign country and got zero results. It's going to be quite complex, esp for non-English languages.

cma 4y ago
This seems to show with scale one-shot stuff like hearing a slang word substitution one time can be done (within the other limitations of transformers) https://arxiv.org/abs/2303.03846 https://arxiv.org/abs/2303.03846

calf 4y ago

Logic is just a different level of abstraction, and the current generation of AIs lack it. Think of it as trying to implement an algorithm in machine code versus writing it in C++ or Haskell. Neural networks are analogous to low-level machine code. Chomsky is saying that that's a reductive and insightful and inefficient way of doing things. Human intelligence works on a higher level of abstraction--even though at the same time our brain cells are literally low-level computational abstractions as well. The two levels coexist in humans, whereas in ChatGPT it took great lengths to produce some semblance of that linguistic structure. Chomsky explains his argument differently but his point is that humans have access to a higher level of computational abstraction, and that is the source of efficiency (we don't need to be trained on thousands of cat pictures for this information-theoretic reason).