4 ms·

"no major AI technology breakthroughs in decades.everything we are seeing is larger compute scaling." This is false. Everything from the transformer to advancem

"no major AI technology breakthroughs in decades.everything we are seeing is larger compute scaling." This is false. Everything from the transformer to advancements in state space models have been foundational breakthroughs

dvt 3y ago
I beg to differ. Transformers are purely an optimization. It’s not exactly right to call everything “compute scaling” but we are still, at the end of the day, fitting polynomials. And frankly, that’s probably not what our brains are doing.
- mitthrowaway2 3y ago
  > And frankly, that’s probably not what our brains are doing. I think that it is! It's much more likely to me that our brains are doing something big and simple than small and complicated. That's the way that nature tends to work. Fitting low-order million-dimensional polynomials would meet that description.
  - deleted 3y ago
    [deleted]
  - dvt 3y ago
    > That's the way that nature tends to work From the double slit experiment, to particle-wave duality, to the particle zoo of the 70s, to quantum chromodynamics, to asymptotic freedom, to more exotic theories like string theory, etc. tells us the complete opposite. Every major discovery in physics in the past 150 years seems to disagree. Things are extremely weird and complicated when we get extremely tiny. Why would our brains be different?
    - mitthrowaway2 3y ago
      If we lived at the quantum scale, then classical physics would be the weird one. Quantum chromodynamics is only confusing for two reasons: it differs from our everyday experience so we don't have an intuition for it, and because it has a large number of mutually-interacting (but basic) components. Richard Feynman put it very well: "The world is strange, the whole universe is very strange, but see when you look at the details then you find out that the rules are very simple, of the game, the mechanical rules by which you can figure out exactly what's going to happen when the situation is simple. It's again this chess game; if you're in just the corner with only a few pieces involved, you can work out exactly what's going to happen. And you can always do that when there's only a few pieces. And so you know you understand it. And yet, in the real game there's so many pieces you can't figure out what's going to happen. "There's such a lot in the world, there's so much distance between the fundamental rules and the final phenomena that it's almost unbelievable that the final variety of phenomena can come from such a steady operation of such simple rules... But it is not complicated, it's just a lot of it."
  - southernplaces7 3y ago
    Our brain is however the very definition of small and complicated. It's the most complex known organic object known to humans and for all that any given normal brain handles in a given day, it uses just 0.3 kilowatt hours (kWh) to do it. No computer we have comes close to handling what a brain handles with that power consumption. ChatGPT by contrast, consumes roughly a gigawatt hour per day serving its users. Yes, to do this it's handling a colossal amount of queries, but that's all it's doing. Your brain handles everything in your body and consciousness, in ways we don't even fully understand, while also letting you think and communicate and reason as a conscious being with self direction. Moreover, there is evidence that at least part of our brain's functions may be exactly as the other reply here mentions, weird, subatomic and deeply complex in ways that are difficult to get a clear grip on.
- f6v 3y ago
  It’s difficult to comprehend impact of something that’s widely adopted. Like, batch normalization alone was probably mind blowing when it came out. Yet it seems so simple and self-explanatory now.
  - imtringued 3y ago
    Or residual connections. It is just addition.
- bjornsing 3y ago
  It is a bit weird though… I mean, you could just as well say that the last breakthrough in all of computing was the transistor (Shockley 1946) – all we’ve been doing ever since is just “scaling” and combining them in new ways.
  - namaria 3y ago
    AI has been a 'moving aim' as opposed to moving target. It's a label slapped on whatever slice of software engineering seeming most magic-like at any given decade. Early computers in late 40s were called 'electronic brains' by the media...
chii 3y ago
People who say things like "major breakthroughs" often imagine a cliff/steep rise. The reality is that most "breakthroughs" are small, incremental, almost invisible steps in all aspects of the field, and potentially even in fields that are only tangentially related. Then, one day, they hear about it in the news, because there's now some hype, or some event that makes it newsworthy. This makes it feel like the breakthrough was instantaneous or steep, but in fact has been in the making for decades or even more. I have no doubt AGI is coming, but it will be gradual and slow. It will be the accumulation of more advances in everything, including hardware, as well as software. It might even include economic changes.
- akasakahakada 3y ago
  From nothing to ChatGPT opened to the public and billions students' lifes rely on that in the next month, isn't this the biggest step in the tech history?
  - smoldesu 3y ago
    > From nothing to ChatGPT There were hundreds of competitive, even SOTA LLMs before ChatGPT existed. You're basically just proving the parent comment right in how small of a leap ChatGPT is from t5-flan or BERT.
  - cykros 3y ago
    Billions of students? Curious about how many kids in India and China are relying on it. I'll accept the prospect of hundreds of millions, generously. And "relying on it" is a strong phrase. Using it as a curiosity, sure. The ones relying on it seem to keep ending up in the news because of how, well, unreliable it is.
  - asadotzler 3y ago
    no.
  - namaria 3y ago
    It's a major societal failure that education has been reduced to turning in coherent enough strings of characters, that's what I've learned in the past year.

dvt 3y ago

I beg to differ. Transformers are purely an optimization. It’s not exactly right to call everything “compute scaling” but we are still, at the end of the day, fitting polynomials. And frankly, that’s probably not what our brains are doing.

mitthrowaway2 3y ago
> And frankly, that’s probably not what our brains are doing. I think that it is! It's much more likely to me that our brains are doing something big and simple than small and complicated. That's the way that nature tends to work. Fitting low-order million-dimensional polynomials would meet that description.
- deleted 3y ago
  [deleted]
- dvt 3y ago
  > That's the way that nature tends to work From the double slit experiment, to particle-wave duality, to the particle zoo of the 70s, to quantum chromodynamics, to asymptotic freedom, to more exotic theories like string theory, etc. tells us the complete opposite. Every major discovery in physics in the past 150 years seems to disagree. Things are extremely weird and complicated when we get extremely tiny. Why would our brains be different?
  - mitthrowaway2 3y ago
    If we lived at the quantum scale, then classical physics would be the weird one. Quantum chromodynamics is only confusing for two reasons: it differs from our everyday experience so we don't have an intuition for it, and because it has a large number of mutually-interacting (but basic) components. Richard Feynman put it very well: "The world is strange, the whole universe is very strange, but see when you look at the details then you find out that the rules are very simple, of the game, the mechanical rules by which you can figure out exactly what's going to happen when the situation is simple. It's again this chess game; if you're in just the corner with only a few pieces involved, you can work out exactly what's going to happen. And you can always do that when there's only a few pieces. And so you know you understand it. And yet, in the real game there's so many pieces you can't figure out what's going to happen. "There's such a lot in the world, there's so much distance between the fundamental rules and the final phenomena that it's almost unbelievable that the final variety of phenomena can come from such a steady operation of such simple rules... But it is not complicated, it's just a lot of it."
- southernplaces7 3y ago
  Our brain is however the very definition of small and complicated. It's the most complex known organic object known to humans and for all that any given normal brain handles in a given day, it uses just 0.3 kilowatt hours (kWh) to do it. No computer we have comes close to handling what a brain handles with that power consumption. ChatGPT by contrast, consumes roughly a gigawatt hour per day serving its users. Yes, to do this it's handling a colossal amount of queries, but that's all it's doing. Your brain handles everything in your body and consciousness, in ways we don't even fully understand, while also letting you think and communicate and reason as a conscious being with self direction. Moreover, there is evidence that at least part of our brain's functions may be exactly as the other reply here mentions, weird, subatomic and deeply complex in ways that are difficult to get a clear grip on.
f6v 3y ago
It’s difficult to comprehend impact of something that’s widely adopted. Like, batch normalization alone was probably mind blowing when it came out. Yet it seems so simple and self-explanatory now.
- imtringued 3y ago
  Or residual connections. It is just addition.
bjornsing 3y ago
It is a bit weird though… I mean, you could just as well say that the last breakthrough in all of computing was the transistor (Shockley 1946) – all we’ve been doing ever since is just “scaling” and combining them in new ways.
- namaria 3y ago
  AI has been a 'moving aim' as opposed to moving target. It's a label slapped on whatever slice of software engineering seeming most magic-like at any given decade. Early computers in late 40s were called 'electronic brains' by the media...

chii 3y ago

People who say things like "major breakthroughs" often imagine a cliff/steep rise. The reality is that most "breakthroughs" are small, incremental, almost invisible steps in all aspects of the field, and potentially even in fields that are only tangentially related. Then, one day, they hear about it in the news, because there's now some hype, or some event that makes it newsworthy. This makes it feel like the breakthrough was instantaneous or steep, but in fact has been in the making for decades or even more. I have no doubt AGI is coming, but it will be gradual and slow. It will be the accumulation of more advances in everything, including hardware, as well as software. It might even include economic changes.

akasakahakada 3y ago
From nothing to ChatGPT opened to the public and billions students' lifes rely on that in the next month, isn't this the biggest step in the tech history?
- smoldesu 3y ago
  > From nothing to ChatGPT There were hundreds of competitive, even SOTA LLMs before ChatGPT existed. You're basically just proving the parent comment right in how small of a leap ChatGPT is from t5-flan or BERT.
- cykros 3y ago
  Billions of students? Curious about how many kids in India and China are relying on it. I'll accept the prospect of hundreds of millions, generously. And "relying on it" is a strong phrase. Using it as a curiosity, sure. The ones relying on it seem to keep ending up in the news because of how, well, unreliable it is.
- asadotzler 3y ago
  no.
- namaria 3y ago
  It's a major societal failure that education has been reduced to turning in coherent enough strings of characters, that's what I've learned in the past year.