4 ms·

My father and his friends were academic computer scientists working on AI back in the 60s. I don't know that there's a straightforward path between what they we

My father and his friends were academic computer scientists working on AI back in the 60s. I don't know that there's a straightforward path between what they were doing and the popular LLMs today, but I do applaud more stories on what old school comp sci researchers were up to.

empath-nirvana 3y ago
There's some value in putting a flag in the ground. Even if most of those people there were in the symbolic camp, a lot of their critiques of neural networks as they existed were well-founded and were really only proved obviously _wrong_ after many many rounds of moore's law.
- marcosdumay 3y ago
  The criticism from the beginning was of a fundamental theoretical nature, and died at the 90's when people proved and demonstrated that neural networks were powerful enough to run any kind of computation. In fact, I don't recall people criticizing neural networks from being too small to be useful. Ever. There was a lot of disagreement between wide and deep network proponents, that deep won by demonstration, but "how large a network we need to handle X" was always more of a question than a "see, we'll never get there". (Even more because the "we will never get there" is obviously false, since the thing practically no limit on scaling.)
  - deleted 3y ago
    [deleted]
fnordpiglet 3y ago
LLMs of today display amazing abductive abilities but are limited in inductive and deductive abilities, as well as other optimization techniques of classical AI and algorithms. These abductive abilities are unique and exciting because we’ve typically done really poorly with ambiguous and complex semantic spaces like this. However I think the excitement has obscured the fact it’s just a piece of a larger machine. Why do we care that LLMs are mediocre chess players when we have machine models using more traditional techniques that are the best chess players on earth? Why do we care they fail at deductive reasoning tests? At mathematical calculations? Those are really well understood areas of computing. Somehow people have fixated on the things we’ve already done that this new technique fails at, but ignore the abilities LLMs and other generative models demonstrate we’ve never achieved before. At the same time the other camp only sees generative AI as the silver bullet tool to end all other tools. Neither is correct.
- dr_dshiv 3y ago
  How are LLMs bad at induction? I thought they were great at induction. This paper doesn’t go into measurements of it, but helps lay out the nature of reasoning well. https://aclanthology.org/2023.findings-acl.67.pdf#page15 https://aclanthology.org/2023.findings-acl.67.pdf#page15
  - einpoklum 3y ago
    They are great at saying things that sounds like the next line of the conversation. That's a certain kind of induction for sure, but probably not the kind you're after.
    - robotresearcher 3y ago
      'Duck induction', by analogy to 'duck typing'.
    - astrange 3y ago
      There's some effective ways to make them a lot more "truthful" than that, as you can tell by how good GPT-4 is at it. Simple example: sample it a bunch of times. If it's different each time, it likely made it up.
- famouswaffles 3y ago
  >but are limited in inductive and deductive abilities LLMs are great at induction. In a broad sense, they are also very good at deduction. "I define a new word, the podition. A podition is any object that can fit on a podium. Is a computer a podition ? Why ?" A correct answer is deductive. LLMs eat these kind of questions for breakfast. Even the OG 2020 GPT-3 could manage them. You really do have to stretch deduction to heights most people struggle with to have them falter majorly.
  - fnordpiglet 3y ago
    This is actually abduction. Induction doesn’t fail, abduction is probabilistic.

empath-nirvana 3y ago

There's some value in putting a flag in the ground. Even if most of those people there were in the symbolic camp, a lot of their critiques of neural networks as they existed were well-founded and were really only proved obviously _wrong_ after many many rounds of moore's law.

marcosdumay 3y ago
The criticism from the beginning was of a fundamental theoretical nature, and died at the 90's when people proved and demonstrated that neural networks were powerful enough to run any kind of computation. In fact, I don't recall people criticizing neural networks from being too small to be useful. Ever. There was a lot of disagreement between wide and deep network proponents, that deep won by demonstration, but "how large a network we need to handle X" was always more of a question than a "see, we'll never get there". (Even more because the "we will never get there" is obviously false, since the thing practically no limit on scaling.)
- deleted 3y ago
  [deleted]

fnordpiglet 3y ago

LLMs of today display amazing abductive abilities but are limited in inductive and deductive abilities, as well as other optimization techniques of classical AI and algorithms. These abductive abilities are unique and exciting because we’ve typically done really poorly with ambiguous and complex semantic spaces like this. However I think the excitement has obscured the fact it’s just a piece of a larger machine. Why do we care that LLMs are mediocre chess players when we have machine models using more traditional techniques that are the best chess players on earth? Why do we care they fail at deductive reasoning tests? At mathematical calculations? Those are really well understood areas of computing. Somehow people have fixated on the things we’ve already done that this new technique fails at, but ignore the abilities LLMs and other generative models demonstrate we’ve never achieved before. At the same time the other camp only sees generative AI as the silver bullet tool to end all other tools. Neither is correct.

dr_dshiv 3y ago
How are LLMs bad at induction? I thought they were great at induction. This paper doesn’t go into measurements of it, but helps lay out the nature of reasoning well. https://aclanthology.org/2023.findings-acl.67.pdf#page15 https://aclanthology.org/2023.findings-acl.67.pdf#page15
- einpoklum 3y ago
  They are great at saying things that sounds like the next line of the conversation. That's a certain kind of induction for sure, but probably not the kind you're after.
  - robotresearcher 3y ago
    'Duck induction', by analogy to 'duck typing'.
  - astrange 3y ago
    There's some effective ways to make them a lot more "truthful" than that, as you can tell by how good GPT-4 is at it. Simple example: sample it a bunch of times. If it's different each time, it likely made it up.
famouswaffles 3y ago
>but are limited in inductive and deductive abilities LLMs are great at induction. In a broad sense, they are also very good at deduction. "I define a new word, the podition. A podition is any object that can fit on a podium. Is a computer a podition ? Why ?" A correct answer is deductive. LLMs eat these kind of questions for breakfast. Even the OG 2020 GPT-3 could manage them. You really do have to stretch deduction to heights most people struggle with to have them falter majorly.
- fnordpiglet 3y ago
  This is actually abduction. Induction doesn’t fail, abduction is probabilistic.