7 ms·

Interesting article, but the flourish ending """AI will soon (if not already) increase in abilities at an exponential rate.""" is not at all substantiated. Woul

Interesting article, but the flourish ending """AI will soon (if not already) increase in abilities at an exponential rate.""" is not at all substantiated. Would be nice to know how the author gets to that conclusion.

ioulaum 2y ago
The exponential part may be iffy, but it is self improving. And this same RL is also creating improvements in small model performance. So, more LLMs are about to rise in quality.
- immibis 2y ago
  It's self-improving? So, we can ask AI how to improve AI, and the suggestions actually work?
  - sesm 2y ago
    It's more like Intel in early days using their CPUs to compute layout for bigger CPUs.
    - ethbr1 2y ago
      Effectively: is the limiting factor to improvement addressable by the thing being improved? If yes, then you get exponential increases very trivially. If no, then something external continues to bottleneck progress.
FridgeSeal 2y ago
It’s basically the line for all the AI-hype people: “all the problems are going away!”, “soon it’ll all magically make things exponentially good-er-er!”
- brookst 2y ago
  Alternatively, it’s a restatement of the obvious empirical truth that technology tends in improve on an exponential and not linear curve. Seems like a simpler explanation that doesn’t even require insulting people.
  - Yossarrian22 2y ago
    The premise would be better supported if it could be shown that if we could 10x the speed at which matrix multiplication is performed conferred a linear or better increase in performance post GPT-4. As it stands that would just seem to give us current results faster, not better results
    - brookst 2y ago
      Efficiency matters but it took semiconductors decades to care about it. Why would it be different this time around?
  - deleted 2y ago
    [deleted]
  - suddenlybananas 2y ago
    That's why airplanes are the so much faster than they were 20 years ago.
  - einsteinx2 2y ago
    I would argue that any given technology tends to improve on an S curve, so exponentially at first and then flattening out. See Moore’s law as a great example. Or more on topic see the improvements in LLMs since they were invented. At first each release was an order of magnitude better than the last (see GPT 2 vs 3 vs 4), now they’re getting better but at a much slower rate. Certainly feels like being at the top of an S curve to me, at least until an entirely new architecture is invented to supersede transformers.
sandworm101 2y ago
The key "ability" that will grow exponentially is AIs ability to convert investment dollars into silicon+electricity and then further reduce those into heat energy. Such schemes only seem wasteful to outsiders, those whose salaries are not tied to their ability to convert money into heat. A fun startup would be one that generates useful electricity from the AI investment cycle. If we put the Ai machine under a pot of water, we might then use the resulting steam to drive a turbine.
- grues-dinner 2y ago
  Some datacentres do in fact recover the heat for things like municipal heating. It's tricky though because being near population centres that can use the heat is often (not always) inversely related to things that are good for datacentres like cheap land, power and lack of neighbours to moan about things like construction and cooling system noise. There was also a startup selling/renting bitcoin miners that doubled as electrical heaters. The problem is that computers are fundamentally resistors, so at most you can get 100% of the energy back as heat. But a heat pump can give you 2-4 times the energy back. So your AI work (or bitcoin mining) plus the capital outlay of the expensive computers has to be worth the difference.
- immibis 2y ago
  Due to Carnot's law, you can't get much electricity that way without a big temperature difference. Think about it: the AI machine would have to run at at least 100 degrees Celsius to boil the water, and that's the bare minimum. But if we can make computers that run at, say, 2000 degrees, without using several times more electricity, then we can capture their waste heat and turn a big portion of it back into electricity to re-feed the computers. It doesn't violate thermodynamics, it's just an alternative possibility to make more computers that use less electricity overall (an alternative to directly trying to reduce the energy usage of silicon logic gates) as long as we're still well above Landauer's limit.
  - sandworm101 2y ago
    At sea level. Put the machine in a vacuum chamber, or atop a big mountain, and we will boil the Ai kettles at less than 100c.
tkellogg 2y ago
Author here. I do believe it's going to be exponential (not yet), but that's out of scope for the article. However, if someone has a good explainer link for that, please put it here and I'll link it into the post.
- jcgrillo 2y ago
  Care to justify those beliefs or are we just supposed to trust your intuition? Why exponential and not merely quadratic (or some other polynomial)? How do you even quantify "it"? I'm teasing, somewhat, because I don't actually expect you're able to answer. Yours isn't reasoned arguments, merely religious fervor dressed up in techy garb. Prove me wrong!
  - globalnode 2y ago
    that seems a bit harsh dont you think? besides youre the one making the assertion, you kinda need to do the proving ;)
    - jcgrillo 2y ago
      No, I don't think it's overly harsh. This hype is out of control and it's important to push back on breathless "exponential" nonsense. That's a term with well defined easily demonstrated mathematical meaning. If you're going to claim growth in some quantity x is exponential, show me that measurements of that quantity fit an exponential function (as opposed to some other function) or provide me a falsifiable theory predicting said fit.
      - kingkongjaffa 2y ago
        I believe they are using 'exponential' as a colloquialism rather than a strict mathematical definition. That aside, we would need to see some evidence of AI developments being bootstrapped by the previous SOTA model as key part of building the next model. For now, it's still human researchers pushing the SOTA models forwards. When people use the term exponential I feel that what they really mean is 'making something so _good_ that it can be used to make the N+1 iteration _more good_ than the last.
the8472 2y ago
The drumbeat of AI progress has been fairly steady, on log scales. https://time.com/6300942/ai-progress-charts/ https://time.com/6300942/ai-progress-charts/
- amazingamazing 2y ago
  that doesn't mean ai is improving itself though
  - the8472 2y ago
    My point was that it already was on an exponential trajectory. RL/self-play and the like remove some of the human inputs that were previously required for that growth. Take the trajectory of chess. handcrafted rules -> policies based on human game statistics -> self-play bootstrapped from human games -> random-initialized self-play.
    - pdimitar 2y ago
      Chess is a game with complete information. Not a good analogy with the real world.
JoshTko 2y ago
AI will improve at an exponential rate once it can independently improve AI performance. For example. Once AI can organically identify, test, confirm, deploy an improvement like R1 vs o1 (in terms of perf/watt) then we'll see exponential improvement. Honestly though, that still seems possible within 5 years or less, maybe 3.
- amelius 2y ago
  Only if the AI can do it faster than humans. And if the improvements it makes are not asymptotically diminishing.
- ksec 2y ago
  >Honestly though, that still seems possible within 5 years or less, maybe 3. If that is a normal human estimation I would guess in reality it is more likely to be in 6-10 years. Which is still good if we get it in 2030 - 2035.
  - ethbr1 2y ago
    For futurism on things that promise economic rewards, exponential increases are not uncommon.
dfilppi 2y ago
[dead]
boznz 2y ago
Currently AI is getting better at sorting the data that already exists, but if enough reddit, wiki posts are wrong its answer is inevitably wrong. Without being able to experiment to test its theories against reality, the AI curve will likely not lead to super-intelligence without humans to assist. That's my 5 cents.

ioulaum 2y ago

The exponential part may be iffy, but it is self improving. And this same RL is also creating improvements in small model performance. So, more LLMs are about to rise in quality.

immibis 2y ago
It's self-improving? So, we can ask AI how to improve AI, and the suggestions actually work?
- sesm 2y ago
  It's more like Intel in early days using their CPUs to compute layout for bigger CPUs.
  - ethbr1 2y ago
    Effectively: is the limiting factor to improvement addressable by the thing being improved? If yes, then you get exponential increases very trivially. If no, then something external continues to bottleneck progress.

FridgeSeal 2y ago

It’s basically the line for all the AI-hype people: “all the problems are going away!”, “soon it’ll all magically make things exponentially good-er-er!”

brookst 2y ago
Alternatively, it’s a restatement of the obvious empirical truth that technology tends in improve on an exponential and not linear curve. Seems like a simpler explanation that doesn’t even require insulting people.
- Yossarrian22 2y ago
  The premise would be better supported if it could be shown that if we could 10x the speed at which matrix multiplication is performed conferred a linear or better increase in performance post GPT-4. As it stands that would just seem to give us current results faster, not better results
  - brookst 2y ago
    Efficiency matters but it took semiconductors decades to care about it. Why would it be different this time around?
- deleted 2y ago
  [deleted]
- suddenlybananas 2y ago
  That's why airplanes are the so much faster than they were 20 years ago.
- einsteinx2 2y ago
  I would argue that any given technology tends to improve on an S curve, so exponentially at first and then flattening out. See Moore’s law as a great example. Or more on topic see the improvements in LLMs since they were invented. At first each release was an order of magnitude better than the last (see GPT 2 vs 3 vs 4), now they’re getting better but at a much slower rate. Certainly feels like being at the top of an S curve to me, at least until an entirely new architecture is invented to supersede transformers.

sandworm101 2y ago

The key "ability" that will grow exponentially is AIs ability to convert investment dollars into silicon+electricity and then further reduce those into heat energy. Such schemes only seem wasteful to outsiders, those whose salaries are not tied to their ability to convert money into heat. A fun startup would be one that generates useful electricity from the AI investment cycle. If we put the Ai machine under a pot of water, we might then use the resulting steam to drive a turbine.

grues-dinner 2y ago
Some datacentres do in fact recover the heat for things like municipal heating. It's tricky though because being near population centres that can use the heat is often (not always) inversely related to things that are good for datacentres like cheap land, power and lack of neighbours to moan about things like construction and cooling system noise. There was also a startup selling/renting bitcoin miners that doubled as electrical heaters. The problem is that computers are fundamentally resistors, so at most you can get 100% of the energy back as heat. But a heat pump can give you 2-4 times the energy back. So your AI work (or bitcoin mining) plus the capital outlay of the expensive computers has to be worth the difference.
immibis 2y ago
Due to Carnot's law, you can't get much electricity that way without a big temperature difference. Think about it: the AI machine would have to run at at least 100 degrees Celsius to boil the water, and that's the bare minimum. But if we can make computers that run at, say, 2000 degrees, without using several times more electricity, then we can capture their waste heat and turn a big portion of it back into electricity to re-feed the computers. It doesn't violate thermodynamics, it's just an alternative possibility to make more computers that use less electricity overall (an alternative to directly trying to reduce the energy usage of silicon logic gates) as long as we're still well above Landauer's limit.
- sandworm101 2y ago
  At sea level. Put the machine in a vacuum chamber, or atop a big mountain, and we will boil the Ai kettles at less than 100c.

tkellogg 2y ago

Author here. I do believe it's going to be exponential (not yet), but that's out of scope for the article. However, if someone has a good explainer link for that, please put it here and I'll link it into the post.

jcgrillo 2y ago
Care to justify those beliefs or are we just supposed to trust your intuition? Why exponential and not merely quadratic (or some other polynomial)? How do you even quantify "it"? I'm teasing, somewhat, because I don't actually expect you're able to answer. Yours isn't reasoned arguments, merely religious fervor dressed up in techy garb. Prove me wrong!
- globalnode 2y ago
  that seems a bit harsh dont you think? besides youre the one making the assertion, you kinda need to do the proving ;)
  - jcgrillo 2y ago
    No, I don't think it's overly harsh. This hype is out of control and it's important to push back on breathless "exponential" nonsense. That's a term with well defined easily demonstrated mathematical meaning. If you're going to claim growth in some quantity x is exponential, show me that measurements of that quantity fit an exponential function (as opposed to some other function) or provide me a falsifiable theory predicting said fit.
    - kingkongjaffa 2y ago
      I believe they are using 'exponential' as a colloquialism rather than a strict mathematical definition. That aside, we would need to see some evidence of AI developments being bootstrapped by the previous SOTA model as key part of building the next model. For now, it's still human researchers pushing the SOTA models forwards. When people use the term exponential I feel that what they really mean is 'making something so _good_ that it can be used to make the N+1 iteration _more good_ than the last.

the8472 2y ago

The drumbeat of AI progress has been fairly steady, on log scales. https://time.com/6300942/ai-progress-charts/ https://time.com/6300942/ai-progress-charts/

amazingamazing 2y ago
that doesn't mean ai is improving itself though
- the8472 2y ago
  My point was that it already was on an exponential trajectory. RL/self-play and the like remove some of the human inputs that were previously required for that growth. Take the trajectory of chess. handcrafted rules -> policies based on human game statistics -> self-play bootstrapped from human games -> random-initialized self-play.
  - pdimitar 2y ago
    Chess is a game with complete information. Not a good analogy with the real world.

JoshTko 2y ago

AI will improve at an exponential rate once it can independently improve AI performance. For example. Once AI can organically identify, test, confirm, deploy an improvement like R1 vs o1 (in terms of perf/watt) then we'll see exponential improvement. Honestly though, that still seems possible within 5 years or less, maybe 3.

amelius 2y ago
Only if the AI can do it faster than humans. And if the improvements it makes are not asymptotically diminishing.
ksec 2y ago
>Honestly though, that still seems possible within 5 years or less, maybe 3. If that is a normal human estimation I would guess in reality it is more likely to be in 6-10 years. Which is still good if we get it in 2030 - 2035.
- ethbr1 2y ago
  For futurism on things that promise economic rewards, exponential increases are not uncommon.

dfilppi 2y ago

[dead]

boznz 2y ago

Currently AI is getting better at sorting the data that already exists, but if enough reddit, wiki posts are wrong its answer is inevitably wrong. Without being able to experiment to test its theories against reality, the AI curve will likely not lead to super-intelligence without humans to assist. That's my 5 cents.