7 ms·

So what happens when they achieve AGI? Will a benevolent network of vastly smarter-than-human intelligences insist on maintaining the wealth hierarchies that hu

So what happens when they achieve AGI? Will a benevolent network of vastly smarter-than-human intelligences insist on maintaining the wealth hierarchies that humans had before AGI arrived? Isn't the point of AGI removing scarcity? I worry that those who became billionaires in the AI boom won't want the relative status of their wealth to become moot once AGI hits. Most likely this will come in the form of artificial barriers to using AI that, for ostensible safety reasons, makes it prohibitively difficult for all but the wealthiest or AGI-lab adjacent social circles to use. This will cause a natural exacerbation of the existing wealth disparities, as if you have access to a smarter AI than everyone else, you can leverage your compute to be tactically superior in any domain with a reward. All we can hope for is a general benevolence and popular consensus that avoids a runaway race to the bottom effect as a result of all this.

catmanjan 1y ago
How can anyone still believe the AGI scam
- atleastoptimal 1y ago
  If you think the possibility of AGI within 7-10 years is a scam then you aren't paying attention to trends.
  - godelski 1y ago
    I wouldn't call 7-10 years a scam, but I would call it low odds. It is pretty hard to be accurate on predictions of a 10 year window. But I definitely think 2027 and 2030 predictions are a scam. Majority of researchers think it is further away than 10 years, if you are looking at surveys from the AI conferences rather than predictions in the news.
    - atleastoptimal 1y ago
      The thing is, AI researchers have continually underestimated the pace of AI progress https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/ https://80000hours.org/2025/03/when-do-experts-expect-agi-to... >One way to reduce selection effects is to look at a wider group of AI researchers than those working on AGI directly, including in academia. This is what Katja Grace did with a survey of thousands of recent AI publication authors. >In 2022, they thought AI wouldn’t be able to write simple Python code until around 2027. >In 2023, they reduced that to 2025, but AI could maybe already meet that condition in 2023 (and definitely by 2024). >Most of their other estimates declined significantly between 2023 and 2022. >The median estimate for achieving ‘high-level machine intelligence’ shortened by 13 years. Basically every median timeline estimate has shrunk like clockwork every year. Back in 2021 people thought it wouldn't be until 2040 or so when AI models could look at a photo and give a human-level textual description of its contents. I think is reasonable to expect that the pace of "prediction error" won't change significantly since it's been on a straight downward trend over the past 4 years, and if it continues as such, AGI around 2028-2030 is a median estimate.
      - parineum 1y ago
        Those are differences of magnitude, AGI is a difference of kind. No amount of describing pictures in natural language is AGI.
        atleastoptimal 1y ago
        I never said it was sufficient for AGI, just that it was a milestone in AI that people thought was farther off than it turned out to be. This is applying to all subsets of intelligence AI is reaching earlier than experts initially predicted, giving good reason AGI (perhaps a synthesis of these elements coming together in a single model, or a suite of models) is likely closer than standard expert consensus.
        parineum 1y ago
        The milestones your citing are all milestones of transformers that were underestimated. If you think an incremental improvement in transformers are what's needed for AGI, I see your angle. However, IMO, transformers haven't shown any evidence of that capability. I see no reason to believe that they'd develop that with a bit more compute or a bit more data.
  - catmanjan 1y ago
    Even if we spent 1 million years on LLM it will not result in AGI, we are no closer to AGI with LLM technology than we were with toaster technology
    - exasperaited 1y ago
      “Would you like a toasted teacake?”
  - BeFlatXIII 1y ago
    Paying attention to trends is how you lose your money on a hype train.
- jimmy2times 1y ago
  I can't believe this is so unpopular here. Maybe it's the tone, but come on, how do people rationally extrapolate from LLMs or even large multimodal generative models to "general intelligence"? Sure, they might do a better job than the average person on a range of tasks, but they're always prone to funny failures pretty much by design (train vs test distribution mismatch). They might combine data in interesting ways you hadn't thought of; that doesn't mean you can actually rely on them in the way you do on a truly intelligent human.
  - wulfstan 1y ago
    I think it’s selection bias - a y-combinator forum is going to have a larger percentage of people who are techno-utopianists than general society, and there will be many seeking financial success by connecting with a trend at the right moment. It seems obvious to me that LLMs are interesting but not revolutionary, and equally obvious that they aren’t heading for any kind of “general intelligence”. They’re good at pretending, and only good at that to the extent that they can mine what has already been expressed. I suppose some are genuine materialists who think that ultimately that is all we are as humans, just a reconstitution of what has come before. I think we’re much more complicated than that. LLMs are like the myth of Narcissus and hypnotically reflect our own humanity back at us.
genedan 1y ago
My hope is that we can at least get a Civ game with a competent AI...

catmanjan 1y ago

How can anyone still believe the AGI scam

atleastoptimal 1y ago
If you think the possibility of AGI within 7-10 years is a scam then you aren't paying attention to trends.
- godelski 1y ago
  I wouldn't call 7-10 years a scam, but I would call it low odds. It is pretty hard to be accurate on predictions of a 10 year window. But I definitely think 2027 and 2030 predictions are a scam. Majority of researchers think it is further away than 10 years, if you are looking at surveys from the AI conferences rather than predictions in the news.
  - atleastoptimal 1y ago
    The thing is, AI researchers have continually underestimated the pace of AI progress https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/ https://80000hours.org/2025/03/when-do-experts-expect-agi-to... >One way to reduce selection effects is to look at a wider group of AI researchers than those working on AGI directly, including in academia. This is what Katja Grace did with a survey of thousands of recent AI publication authors. >In 2022, they thought AI wouldn’t be able to write simple Python code until around 2027. >In 2023, they reduced that to 2025, but AI could maybe already meet that condition in 2023 (and definitely by 2024). >Most of their other estimates declined significantly between 2023 and 2022. >The median estimate for achieving ‘high-level machine intelligence’ shortened by 13 years. Basically every median timeline estimate has shrunk like clockwork every year. Back in 2021 people thought it wouldn't be until 2040 or so when AI models could look at a photo and give a human-level textual description of its contents. I think is reasonable to expect that the pace of "prediction error" won't change significantly since it's been on a straight downward trend over the past 4 years, and if it continues as such, AGI around 2028-2030 is a median estimate.
    - parineum 1y ago
      Those are differences of magnitude, AGI is a difference of kind. No amount of describing pictures in natural language is AGI.
      - atleastoptimal 1y ago
        I never said it was sufficient for AGI, just that it was a milestone in AI that people thought was farther off than it turned out to be. This is applying to all subsets of intelligence AI is reaching earlier than experts initially predicted, giving good reason AGI (perhaps a synthesis of these elements coming together in a single model, or a suite of models) is likely closer than standard expert consensus.
        parineum 1y ago
        The milestones your citing are all milestones of transformers that were underestimated. If you think an incremental improvement in transformers are what's needed for AGI, I see your angle. However, IMO, transformers haven't shown any evidence of that capability. I see no reason to believe that they'd develop that with a bit more compute or a bit more data.
- catmanjan 1y ago
  Even if we spent 1 million years on LLM it will not result in AGI, we are no closer to AGI with LLM technology than we were with toaster technology
  - exasperaited 1y ago
    “Would you like a toasted teacake?”
- BeFlatXIII 1y ago
  Paying attention to trends is how you lose your money on a hype train.
jimmy2times 1y ago
I can't believe this is so unpopular here. Maybe it's the tone, but come on, how do people rationally extrapolate from LLMs or even large multimodal generative models to "general intelligence"? Sure, they might do a better job than the average person on a range of tasks, but they're always prone to funny failures pretty much by design (train vs test distribution mismatch). They might combine data in interesting ways you hadn't thought of; that doesn't mean you can actually rely on them in the way you do on a truly intelligent human.
- wulfstan 1y ago
  I think it’s selection bias - a y-combinator forum is going to have a larger percentage of people who are techno-utopianists than general society, and there will be many seeking financial success by connecting with a trend at the right moment. It seems obvious to me that LLMs are interesting but not revolutionary, and equally obvious that they aren’t heading for any kind of “general intelligence”. They’re good at pretending, and only good at that to the extent that they can mine what has already been expressed. I suppose some are genuine materialists who think that ultimately that is all we are as humans, just a reconstitution of what has come before. I think we’re much more complicated than that. LLMs are like the myth of Narcissus and hypnotically reflect our own humanity back at us.

genedan 1y ago

My hope is that we can at least get a Civ game with a competent AI...