3 ms·

The article literally writes >the Perceptron would lead to machines that “will be able to walk, talk, see, write, reproduce itself and be conscious of its exis

The article literally writes >the Perceptron would lead to machines that “will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” More than six decades later, similar claims are being made about current artificial intelligence. So, what’s changed in the intervening years? In some ways, not much. Which shows the author's complete lack of understanding of the basics. In ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. So what has changed? And why is today fundamentally different, despite us still using the same perceptron architecture with a few tweaks? Look at the Kurzweil curve [1]. The available compute power has allowed existing architectures to flourish beyond what was possible in the 60s. Do you think it's a coincidence that computers in the early 2010s were suddenly able to distinguish pictures of cats and dogs reliably, when that was utterly impossible just a decade earlier? Or perhaps you think it's weird that GPT3/4 capability level models arrived in the 2020s? Just look at the curve (which was made in the early 2000s btw) to find out why. [1] https://images.squarespace-cdn.com/content/v1/5ec0712f88e81651845998f2/1598538084191-81G58RVP0IDI8XIFOW5U/kurzweil-2.fw.png https://images.squarespace-cdn.com/content/v1/5ec0712f88e816...

wizzwizz4 3y ago
> In the ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. Hate to say "no, you're wrong", but… this is just not true. The latest LLM stuff only came after the invention of the transformer architecture. After the development of a new architecture, we always have a "hey, if you throw a supercomputer at it, it works EVEN BETTER!" phase: think Deep Blue, which was just alpha-beta search with a fitted evaluation function on a really powerful machine. I see no reason to believe that transformers doing text prediction are a generally-applicable AI architecture. I suspect they're even incapable of many things, no matter how big they get: GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days.
- sigmoid10 3y ago
  If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still around is because it was actually a step back from recurrent networks and re-allowed scaling once more. Recurrent models used to be the big hope towards AGI for a long time (and some people still cling to them), but they simply turned out to be too limited when you want to scale up. And in the end, it's all about scaling compute. No matter which way you look at it. This is the true constant over the past 60 years. >GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days. Which are...? Anything we have seen so far from scaling LLMs is that capability is purely an issue of training/fine-tuning. It's not a question of the architecture.
  - wizzwizz4 3y ago
    My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.
    - sigmoid10 3y ago
      >You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it Thank you for confirming my point. Because this is what it's all about. It might not seem like much if you only think on timeframes of startups, but in the long run this will beat any architecture improvement. Yeah, maybe you can come up with a fancy revolutionary design and achieve instant 50% improvement over SOTA. You will get all the attention of the media and prizes and stuff. But how often does that happen? On the other hand, you could also just silently wait one or two generations of Nvidia cards and get the same thing. And even better, this also works when there is absolutely no improvement in architectures by anyone. This is only highlighted by transformers, because they came out in 2017, but only now have they been able to solve e.g. the bar exam. Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with.

wizzwizz4 3y ago

> In the ML reasearch, it has become very clear that compute is everything and architectures are actually not that important. Hate to say "no, you're wrong", but… this is just not true. The latest LLM stuff only came after the invention of the transformer architecture. After the development of a new architecture, we always have a "hey, if you throw a supercomputer at it, it works EVEN BETTER!" phase: think Deep Blue, which was just alpha-beta search with a fitted evaluation function on a really powerful machine. I see no reason to believe that transformers doing text prediction are a generally-applicable AI architecture. I suspect they're even incapable of many things, no matter how big they get: GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days.

sigmoid10 3y ago
If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still around is because it was actually a step back from recurrent networks and re-allowed scaling once more. Recurrent models used to be the big hope towards AGI for a long time (and some people still cling to them), but they simply turned out to be too limited when you want to scale up. And in the end, it's all about scaling compute. No matter which way you look at it. This is the true constant over the past 60 years. >GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days. Which are...? Anything we have seen so far from scaling LLMs is that capability is purely an issue of training/fine-tuning. It's not a question of the architecture.
- wizzwizz4 3y ago
  My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.
  - sigmoid10 3y ago
    >You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it Thank you for confirming my point. Because this is what it's all about. It might not seem like much if you only think on timeframes of startups, but in the long run this will beat any architecture improvement. Yeah, maybe you can come up with a fancy revolutionary design and achieve instant 50% improvement over SOTA. You will get all the attention of the media and prizes and stuff. But how often does that happen? On the other hand, you could also just silently wait one or two generations of Nvidia cards and get the same thing. And even better, this also works when there is absolutely no improvement in architectures by anyone. This is only highlighted by transformers, because they came out in 2017, but only now have they been able to solve e.g. the bar exam. Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with.