3 ms·

Compute will always be the bottleneck even if this were true.

by 10xDev 26d ago

Miner49er 26d ago
Eventually recursive self-improvement includes reducing bottlenecks.
- 10xDev 26d ago
  Eventually the bottleneck might be people themselves.
  - ccozan 26d ago
    Improbably, the real bottleneck is energy.
- glenstein 26d ago
  Which is to say, scalable and open-ended capability of ramping up physical infrastructure. I don't know that that's achievable yet. Though the era of increasingly advanced and automated robotics seems to be around the corner which could create a cycle, vicious or virtuous depending on how you feel about it.
Fordec 26d ago
If humans can figure out to optimize to circumvent bottlenecks, I have no doubt each new bottleneck will also get routed around, just now automated.
- dakolli 26d ago
  How do you automate the mines to get the raw materials to make the compute from, and build additional fabs that take a almost a decade to stand up. You're actually delusional.
  - Fordec 26d ago
    Hello good sir from the 1700s pre-industrial revolution who doesn't think that mines and factories can be automated.
    - dakolli 26d ago
      The factories that supply the equipment, maintain the equipment, the energy inputs, the financials of those mines are not automated. People on hackernews are actually some of the dumbest people on the internet. This place is worse than lesswrong.
      - scott_weber 26d ago
        The question of whether something can be automated is distinct from the question of whether it is currently automated. Things can can be automated may transition to being automated in practice in the future as technology improves and investment deepens.
  - a2ff6eeb0 26d ago
    https://en.wikipedia.org/wiki/Lights_out_(manufacturing) https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Scroll down to the existing examples section.
  - 7373737373 26d ago
    Some mines are already heavily automated: https://youtube.com/watch?v=_Z9w-mUoUsY https://youtube.com/watch?v=_Z9w-mUoUsY https://youtube.com/watch?v=SRuht0QIprs https://youtube.com/watch?v=SRuht0QIprs
- mrbungie 26d ago
  We are not in an everything-has-an-API world yet, and it'll for sure take some time to get there.
  - Fordec 26d ago
    For sure. Anyone who thinks that we're in the end state of what progress can be made simply lacks imagination. This is all going to keep changing and iterating for the rest of our natural lives. The only constant is change.
  - combobyte 26d ago
    I'd argue we've been in an "everything-has-an-API" world for a long time now — it's just that discoverability of said APIs is still crap.
    - classified 26d ago
      And since LLMs are apparently good at circumventing the absence of an API, there's not much incentive to add them now. APIs are for humans. LLMs just break through all the captchas and anti-bot measures.
      - jasomill 26d ago
        Humans do this too.
        classified 26d ago
        Yes, but for humans it creates friction. Seeing a captcha makes me think twice and thrice if I really want to visit that site so badly that I'll endure the suckage. LLMs don't care, for them the friction doesn't exist.
- HenrikPontoppid 26d ago
  Yes. In other words: the singularity. I'll only believe it when I see it though.
  - Fordec 26d ago
    I'm coming around to not liking the term singularity, it implies an endpoint or finish line rather than something that just keeps continuing and evolving.
    - supern0va 26d ago
      It doesn't imply that. The singularity is just the inflection point.
      - jsLavaGoat 26d ago
        Singularity and inflection point are incompatible mathematically and in the plain sense, it really is focused on a particular moment and always has been, hence the term. And it's definitely supposed to imply some kind of historical discontinuity not a change in convexity.
      - Fordec 26d ago
        Which assumes the presence of an inflection point that keeps inflecting rather than revert to an S-curve. The growth model is not borne out yet to declare what shape it is.
        supern0va 25d ago
        Certainly. The singularity sort of assumes that there is not a fixed limit to intelligence, or at least that if there is, it's quite a ways away. That may not be true.
    - piloto_ciego 26d ago
      I've done a lot of thinking about this since I first used ChatGPT to write some BS jinja2 templates hours after I first play with it. I said to my friend then (who scoffed at me) that "man, this is incredible, I think we're in the foothills of the singularity! This is insane! Sure it's stupid now but I can't believe this is even possible!" That friend is so black pilled and bitter he now hates AI. Whatever, I can't fix that, but the current progress is astounding. But thinking about the geometry of this problem helps understand why people aren't adjusting well to this. While we're walking on the curve, we look at the rate of change of the curve and say, "well, yeah, of course, dy/dx is 5 at this point and was 1 at the point a few years ago, because the curve is getting steeper" but we're always going to feel this way as things rip off into the stratosphere because dy/dx(e^x) = e^x. From standing on the curve the curve is notably seeper, but the steepness totally makes sense to you. It's only when you look back 10 years or so that you think "wait a second, holy hell, I couldn't have imagined this!" The first time I really (I mean really) thought about the singularity and AI was in roughly 2014. I mean, yea, I'd thought about things before that, but yeah, before the Humans need not apply video, I'd never actually given it much thought. I think back to myself 10 - 15 years or so ago, when I was just starting to tackle real programming projects, and was just starting to get decent at writing code, there is no way on earth that I would have imagined that a little over a decade later, Navier-Stokes would be solved by a computer program and the vast majority of my work would be playing sooth sayer to increasingly complicated piles of linear algebra.
monster_truck 26d ago
Based on the leaps in local inference speed in the past month, which have been absurd, I'm p confident we're going to whiplash from compute constrained to storage constrained. Bit apples to oranges, but it reminds me of all the fiber we installed in the late 90s, certain that per-strand capacity increases were years or decades out, only to get massively rugged
- Fordec 26d ago
  I expect the investments into AI driven mathematic discoveries that underpin compression efficiency will be a key investment area. Particularly at the data center scale rather than per device or per file level.
  - xtracto 26d ago
    pi-fs will solve all our data compression problems.
  - monster_truck 26d ago
    It's not going to be enough. The naive approach of a project I've been working on was pushing >10gbps over the local network, after a ton of work I got it back down under 1... and now it's processing so much more shit that I'm almost past 5 again! It compresses at >3:1 but the latency hit isn't suitable. I get the impression the only reason there is renewed interest in photonics is because DCs are simply out of room (and power) to rack more servers and switches.
    - Fordec 26d ago
      I 100% agree with your impression. For a good while to come there's going to be a bunch of Jevons Paradox to all of this, but adoption of architectural changes like that photonics adoption is exactly the type of adaption to circumvent bottlenecks I'm referring to. We're going to hit hundreds of bottlenecks and each one will inevitably breed new approaches and technology directions. And the forcing function won't be talking about them, but implementing them, seeing who wins and taking lessons.
hgoel 26d ago
As a statement of fact divorced from context, this is of course true, but it's worth putting it in context of what small-medium scale models have been achieving recently. Many of the most recent releases from Chinese labs are almost on par with trillion parameter models from less than a year ago (edit: despite being small enough to usably run on prosumer hardware). It seems clear parameter efficiency can still be improved dramatically. In which case, maybe we don't need as much compute as we might expect. I hesitate to say "to reach a singularity" because it's kind of hard to define how that works out. Even intelligence probably hits some scaling limits eventually (e.g. speed of light related restrictions on how far it can scale, or how quickly it can expand).
lijok 26d ago
And the goalposts move again

Compute will always be the bottleneck even if this were true.

by 10xDev 26d ago

Compute will always be the bottleneck even if this were true.

Miner49er 26d ago

Eventually recursive self-improvement includes reducing bottlenecks.

10xDev 26d ago
Eventually the bottleneck might be people themselves.
- ccozan 26d ago
  Improbably, the real bottleneck is energy.
glenstein 26d ago
Which is to say, scalable and open-ended capability of ramping up physical infrastructure. I don't know that that's achievable yet. Though the era of increasingly advanced and automated robotics seems to be around the corner which could create a cycle, vicious or virtuous depending on how you feel about it.

Fordec 26d ago

If humans can figure out to optimize to circumvent bottlenecks, I have no doubt each new bottleneck will also get routed around, just now automated.

dakolli 26d ago
How do you automate the mines to get the raw materials to make the compute from, and build additional fabs that take a almost a decade to stand up. You're actually delusional.
- Fordec 26d ago
  Hello good sir from the 1700s pre-industrial revolution who doesn't think that mines and factories can be automated.
  - dakolli 26d ago
    The factories that supply the equipment, maintain the equipment, the energy inputs, the financials of those mines are not automated. People on hackernews are actually some of the dumbest people on the internet. This place is worse than lesswrong.
    - scott_weber 26d ago
      The question of whether something can be automated is distinct from the question of whether it is currently automated. Things can can be automated may transition to being automated in practice in the future as technology improves and investment deepens.
- a2ff6eeb0 26d ago
  https://en.wikipedia.org/wiki/Lights_out_(manufacturing) https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Scroll down to the existing examples section.
- 7373737373 26d ago
  Some mines are already heavily automated: https://youtube.com/watch?v=_Z9w-mUoUsY https://youtube.com/watch?v=_Z9w-mUoUsY https://youtube.com/watch?v=SRuht0QIprs https://youtube.com/watch?v=SRuht0QIprs
mrbungie 26d ago
We are not in an everything-has-an-API world yet, and it'll for sure take some time to get there.
- Fordec 26d ago
  For sure. Anyone who thinks that we're in the end state of what progress can be made simply lacks imagination. This is all going to keep changing and iterating for the rest of our natural lives. The only constant is change.
- combobyte 26d ago
  I'd argue we've been in an "everything-has-an-API" world for a long time now — it's just that discoverability of said APIs is still crap.
  - classified 26d ago
    And since LLMs are apparently good at circumventing the absence of an API, there's not much incentive to add them now. APIs are for humans. LLMs just break through all the captchas and anti-bot measures.
    - jasomill 26d ago
      Humans do this too.
      - classified 26d ago
        Yes, but for humans it creates friction. Seeing a captcha makes me think twice and thrice if I really want to visit that site so badly that I'll endure the suckage. LLMs don't care, for them the friction doesn't exist.
HenrikPontoppid 26d ago
Yes. In other words: the singularity. I'll only believe it when I see it though.
- Fordec 26d ago
  I'm coming around to not liking the term singularity, it implies an endpoint or finish line rather than something that just keeps continuing and evolving.
  - supern0va 26d ago
    It doesn't imply that. The singularity is just the inflection point.
    - jsLavaGoat 26d ago
      Singularity and inflection point are incompatible mathematically and in the plain sense, it really is focused on a particular moment and always has been, hence the term. And it's definitely supposed to imply some kind of historical discontinuity not a change in convexity.
    - Fordec 26d ago
      Which assumes the presence of an inflection point that keeps inflecting rather than revert to an S-curve. The growth model is not borne out yet to declare what shape it is.
      - supern0va 25d ago
        Certainly. The singularity sort of assumes that there is not a fixed limit to intelligence, or at least that if there is, it's quite a ways away. That may not be true.
  - piloto_ciego 26d ago
    I've done a lot of thinking about this since I first used ChatGPT to write some BS jinja2 templates hours after I first play with it. I said to my friend then (who scoffed at me) that "man, this is incredible, I think we're in the foothills of the singularity! This is insane! Sure it's stupid now but I can't believe this is even possible!" That friend is so black pilled and bitter he now hates AI. Whatever, I can't fix that, but the current progress is astounding. But thinking about the geometry of this problem helps understand why people aren't adjusting well to this. While we're walking on the curve, we look at the rate of change of the curve and say, "well, yeah, of course, dy/dx is 5 at this point and was 1 at the point a few years ago, because the curve is getting steeper" but we're always going to feel this way as things rip off into the stratosphere because dy/dx(e^x) = e^x. From standing on the curve the curve is notably seeper, but the steepness totally makes sense to you. It's only when you look back 10 years or so that you think "wait a second, holy hell, I couldn't have imagined this!" The first time I really (I mean really) thought about the singularity and AI was in roughly 2014. I mean, yea, I'd thought about things before that, but yeah, before the Humans need not apply video, I'd never actually given it much thought. I think back to myself 10 - 15 years or so ago, when I was just starting to tackle real programming projects, and was just starting to get decent at writing code, there is no way on earth that I would have imagined that a little over a decade later, Navier-Stokes would be solved by a computer program and the vast majority of my work would be playing sooth sayer to increasingly complicated piles of linear algebra.

monster_truck 26d ago

Based on the leaps in local inference speed in the past month, which have been absurd, I'm p confident we're going to whiplash from compute constrained to storage constrained. Bit apples to oranges, but it reminds me of all the fiber we installed in the late 90s, certain that per-strand capacity increases were years or decades out, only to get massively rugged

Fordec 26d ago
I expect the investments into AI driven mathematic discoveries that underpin compression efficiency will be a key investment area. Particularly at the data center scale rather than per device or per file level.
- xtracto 26d ago
  pi-fs will solve all our data compression problems.
- monster_truck 26d ago
  It's not going to be enough. The naive approach of a project I've been working on was pushing >10gbps over the local network, after a ton of work I got it back down under 1... and now it's processing so much more shit that I'm almost past 5 again! It compresses at >3:1 but the latency hit isn't suitable. I get the impression the only reason there is renewed interest in photonics is because DCs are simply out of room (and power) to rack more servers and switches.
  - Fordec 26d ago
    I 100% agree with your impression. For a good while to come there's going to be a bunch of Jevons Paradox to all of this, but adoption of architectural changes like that photonics adoption is exactly the type of adaption to circumvent bottlenecks I'm referring to. We're going to hit hundreds of bottlenecks and each one will inevitably breed new approaches and technology directions. And the forcing function won't be talking about them, but implementing them, seeing who wins and taking lessons.

hgoel 26d ago

As a statement of fact divorced from context, this is of course true, but it's worth putting it in context of what small-medium scale models have been achieving recently. Many of the most recent releases from Chinese labs are almost on par with trillion parameter models from less than a year ago (edit: despite being small enough to usably run on prosumer hardware). It seems clear parameter efficiency can still be improved dramatically. In which case, maybe we don't need as much compute as we might expect. I hesitate to say "to reach a singularity" because it's kind of hard to define how that works out. Even intelligence probably hits some scaling limits eventually (e.g. speed of light related restrictions on how far it can scale, or how quickly it can expand).

lijok 26d ago

And the goalposts move again