3 ms·

This article tries to seem like an enlightened sceptic, but it's actually really cringe because of it's lack of insight into the real progress since then. Then

This article tries to seem like an enlightened sceptic, but it's actually really cringe because of it's lack of insight into the real progress since then. Then again I don't know what people would expect from a politics editor with a liberal arts background.

stuaxo 3y ago
What's their background got to do with it? Edit: actually read the article, and it just seems to note the cyclical nature of AI, which the lay person may not be aware of.
coldtea 3y ago
>but it's actually really cringe because of it's lack of insight into the real progress since then Looks to me to have a good enough understanding of the "real progress since then".
- sigmoid10 3y ago
  The article literally writes >the Perceptron would lead to machines that “will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” More than six decades later, similar claims are being made about current artificial intelligence. So, what’s changed in the intervening years? In some ways, not much. Which shows the author's complete lack of understanding of the basics. In ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. So what has changed? And why is today fundamentally different, despite us still using the same perceptron architecture with a few tweaks? Look at the Kurzweil curve [1]. The available compute power has allowed existing architectures to flourish beyond what was possible in the 60s. Do you think it's a coincidence that computers in the early 2010s were suddenly able to distinguish pictures of cats and dogs reliably, when that was utterly impossible just a decade earlier? Or perhaps you think it's weird that GPT3/4 capability level models arrived in the 2020s? Just look at the curve (which was made in the early 2000s btw) to find out why. [1] https://images.squarespace-cdn.com/content/v1/5ec0712f88e81651845998f2/1598538084191-81G58RVP0IDI8XIFOW5U/kurzweil-2.fw.png https://images.squarespace-cdn.com/content/v1/5ec0712f88e816...
  - wizzwizz4 3y ago
    > In the ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. Hate to say "no, you're wrong", but… this is just not true. The latest LLM stuff only came after the invention of the transformer architecture. After the development of a new architecture, we always have a "hey, if you throw a supercomputer at it, it works EVEN BETTER!" phase: think Deep Blue, which was just alpha-beta search with a fitted evaluation function on a really powerful machine. I see no reason to believe that transformers doing text prediction are a generally-applicable AI architecture. I suspect they're even incapable of many things, no matter how big they get: GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days.
    - sigmoid10 3y ago
      If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still around is because it was actually a step back from recurrent networks and re-allowed scaling once more. Recurrent models used to be the big hope towards AGI for a long time (and some people still cling to them), but they simply turned out to be too limited when you want to scale up. And in the end, it's all about scaling compute. No matter which way you look at it. This is the true constant over the past 60 years. >GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days. Which are...? Anything we have seen so far from scaling LLMs is that capability is purely an issue of training/fine-tuning. It's not a question of the architecture.
      - wizzwizz4 3y ago
        My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.
trash_cat 3y ago
It's not cringe because of lack insight into progress, quite the opposite. It's cringe because the author does not understand what an LLM is and what it can be used for.
- __loam 3y ago
  I'm tired of people saying this when someone expresses a skeptical viewpoint. You don't need a doctorate in statistical computing to have an opinion.
- vouaobrasil 3y ago
  It seems that people who DO understand LLMs are too blinded by the shinyness to say one coherent word about it.
ben_w 3y ago
My perspective is that the "hallucination" problem with LLMs is either the same as or analogous to the Gell-Mann Amnesia effect with newspapers. Outside our own field, they look amazing and insightful; within our field, we focus on the "wet pavements cause rain" level of mistakes that they make.
ordu 3y ago
I agree, article gives just one perspective on the matter, seeking similarities with the past and ignoring differences. But I believe this perspective has right to exist, it gives you priors, then you can incorporate differences to update.
vouaobrasil 3y ago
> Then again I don't know what people would expect from a politics editor with a liberal arts background. I'd expect better from them than from technologists who actually understand the technology, becuase the latter typically have unreserved praise for their creations and complete blindness to the negative effects of technology on society.

stuaxo 3y ago

What's their background got to do with it? Edit: actually read the article, and it just seems to note the cyclical nature of AI, which the lay person may not be aware of.

coldtea 3y ago

>but it's actually really cringe because of it's lack of insight into the real progress since then Looks to me to have a good enough understanding of the "real progress since then".

sigmoid10 3y ago
The article literally writes >the Perceptron would lead to machines that “will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” More than six decades later, similar claims are being made about current artificial intelligence. So, what’s changed in the intervening years? In some ways, not much. Which shows the author's complete lack of understanding of the basics. In ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. So what has changed? And why is today fundamentally different, despite us still using the same perceptron architecture with a few tweaks? Look at the Kurzweil curve [1]. The available compute power has allowed existing architectures to flourish beyond what was possible in the 60s. Do you think it's a coincidence that computers in the early 2010s were suddenly able to distinguish pictures of cats and dogs reliably, when that was utterly impossible just a decade earlier? Or perhaps you think it's weird that GPT3/4 capability level models arrived in the 2020s? Just look at the curve (which was made in the early 2000s btw) to find out why. [1] https://images.squarespace-cdn.com/content/v1/5ec0712f88e81651845998f2/1598538084191-81G58RVP0IDI8XIFOW5U/kurzweil-2.fw.png https://images.squarespace-cdn.com/content/v1/5ec0712f88e816...
- wizzwizz4 3y ago
  > In the ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. Hate to say "no, you're wrong", but… this is just not true. The latest LLM stuff only came after the invention of the transformer architecture. After the development of a new architecture, we always have a "hey, if you throw a supercomputer at it, it works EVEN BETTER!" phase: think Deep Blue, which was just alpha-beta search with a fitted evaluation function on a really powerful machine. I see no reason to believe that transformers doing text prediction are a generally-applicable AI architecture. I suspect they're even incapable of many things, no matter how big they get: GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days.
  - sigmoid10 3y ago
    If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still around is because it was actually a step back from recurrent networks and re-allowed scaling once more. Recurrent models used to be the big hope towards AGI for a long time (and some people still cling to them), but they simply turned out to be too limited when you want to scale up. And in the end, it's all about scaling compute. No matter which way you look at it. This is the true constant over the past 60 years. >GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days. Which are...? Anything we have seen so far from scaling LLMs is that capability is purely an issue of training/fine-tuning. It's not a question of the architecture.
    - wizzwizz4 3y ago
      My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.

trash_cat 3y ago

It's not cringe because of lack insight into progress, quite the opposite. It's cringe because the author does not understand what an LLM is and what it can be used for.

__loam 3y ago
I'm tired of people saying this when someone expresses a skeptical viewpoint. You don't need a doctorate in statistical computing to have an opinion.
vouaobrasil 3y ago
It seems that people who DO understand LLMs are too blinded by the shinyness to say one coherent word about it.

ben_w 3y ago

My perspective is that the "hallucination" problem with LLMs is either the same as or analogous to the Gell-Mann Amnesia effect with newspapers. Outside our own field, they look amazing and insightful; within our field, we focus on the "wet pavements cause rain" level of mistakes that they make.

ordu 3y ago

I agree, article gives just one perspective on the matter, seeking similarities with the past and ignoring differences. But I believe this perspective has right to exist, it gives you priors, then you can incorporate differences to update.

vouaobrasil 3y ago

> Then again I don't know what people would expect from a politics editor with a liberal arts background. I'd expect better from them than from technologists who actually understand the technology, becuase the latter typically have unreserved praise for their creations and complete blindness to the negative effects of technology on society.