3 ms·

We are literally standing on the precipice of agency. After that, it's over. You see us approaching a wall, I see a cliff.

WJW 2y ago
To stick with the metaphor, it's not super unclear if agency is within grasp or on the other side of an abyss. LLMs are definitely an improvement, but it's not at all clear if they can scale to human-level agency. If they reach that, it's even more unclear if they could ever reach superhuman levels given that all their training data is human-level. And finally, we can see from normal human society that it is hardly ever the smartest humans who achieve the most or rise to the highest levels of power. There is no reason to believe that an AI with agency would an inherent "it's over" scenario
- philistine 2y ago
  We kill millions of humans with agency every day on this planet. And none of them immediately die if we stop providing them with electrical power ... well I guess a small amount of them do. Anyway, we'll be fine. If the AI come, can they immediately stop us from growing food? How can an AI prevent my orchard from giving apples next year? Sure, it can mess up Facebook, but at this point that'd be a benefit.
  - WJW 2y ago
    I don't want to be a hater but according to the UN about 61 million humans die per year, so that only comes out to ~167k per day rather than millions. Most of those will die from old age too, rather than "being killed". Your main point is true though, even superhuman AI would have a rough time in actual combat. It's just too dependent on electricity and datacenters with known locations to actually have a chance.
    - labster 2y ago
      I’m sure the superintelligent AI will convince humans to transport its core while plugged into a potato battery. But honestly did the supply chain attack on Lebanese pagers last week teach you nothing? AIs should be great at that.
      - philistine 2y ago
        What? Planting C4 in pagers? Those pagers did not blow up on their own. Hundreds of people needed to put all that explosive in all those devices. Our world is still analogue.
  - tivert 2y ago
    > We kill millions of humans with agency every day on this planet. And none of them immediately die if we stop providing them with electrical power ... well I guess a small amount of them do. Anyway, we'll be fine. If the AI come, can they immediately stop us from growing food? How can an AI prevent my orchard from giving apples next year? Sure, it can mess up Facebook, but at this point that'd be a benefit. 1. IMHO, genocidal apocalypse scenarios like you describe are the wrong way to think about the societal danger of AGI. I think a far more likely outcome, and nearly as disastrous, is AGI eliminating the dependency of capital on labor, leading to a global China Shock on steroids (e.g. unimaginable levels of inequality and wealth concentration, which no level of personal adaptability could overcome). 2. Even in your apocalypse scenario, I think you underestimate the damage that could be done. I don't have an orchard, so I know I'm fucked if urban life-support systems get messed up over large enough area that no aid is coming to my local area afterwards. And a genocidal AI that wasn't blindingly stupid would wait until it had control of enough robots to act in the real world, after which it could agent-orange your orchard.
    - philistine 2y ago
      1. A massive loss of jobs like that would lead to a better redistribution of wealth. I don't believe the US would adequately react, but I trust the rest of the world to protect people from the problem. 2. Which robots?? Until robots can, without any human input, make any sort of other robots, then there is nothing to fear from AGI. Right now they barely walk, and the really big boys can solder one joint real good. That's not enough.
      - tivert 2y ago
        > A massive loss of jobs like that would lead to a better redistribution of wealth... How do you come to that conclusion? I think we'll get massive inequality because it'll be far easier to develop new technology than change the core ideology of capitalist society through some secular process. If a "better redistribution of wealth" were in the cards, it probably should have happened already, and probably cannot happen after the 99% loose most of their economic power by being replaced by automation. If the 1% are forward-thinking, they'll push a UBI scheme to keep the 99% from getting too disruptive before they become irrelevant.
  - bottlepalm 2y ago
    How can you shut down something that can multiply into data centers in different countries around the world? I'm not sure you realize this, but computers control everything. A dumb bug shut down windows computers around the world a month ago. A smart AI could potentially rewrite every piece of software and lock us out of everything. Those medicines your friends and family need to stay alive? Yea the factories that produce them only work if you do what the AI says.
- bottlepalm 2y ago
  > There is no reason to believe... What is happening right now is so obvious that people have been predicting it for over 60 years. It is ingrained in our culture. Everyone knows what happens when the AI becomes smarter than us. If you 'see no reason' then you are looking through a microscope. Lift your head up and look around. Agency isn't black and white, it is a gradient. We already have agency to some degree, and it is improving fast.

WJW 2y ago

To stick with the metaphor, it's not super unclear if agency is within grasp or on the other side of an abyss. LLMs are definitely an improvement, but it's not at all clear if they can scale to human-level agency. If they reach that, it's even more unclear if they could ever reach superhuman levels given that all their training data is human-level. And finally, we can see from normal human society that it is hardly ever the smartest humans who achieve the most or rise to the highest levels of power. There is no reason to believe that an AI with agency would an inherent "it's over" scenario

philistine 2y ago
We kill millions of humans with agency every day on this planet. And none of them immediately die if we stop providing them with electrical power ... well I guess a small amount of them do. Anyway, we'll be fine. If the AI come, can they immediately stop us from growing food? How can an AI prevent my orchard from giving apples next year? Sure, it can mess up Facebook, but at this point that'd be a benefit.
- WJW 2y ago
  I don't want to be a hater but according to the UN about 61 million humans die per year, so that only comes out to ~167k per day rather than millions. Most of those will die from old age too, rather than "being killed". Your main point is true though, even superhuman AI would have a rough time in actual combat. It's just too dependent on electricity and datacenters with known locations to actually have a chance.
  - labster 2y ago
    I’m sure the superintelligent AI will convince humans to transport its core while plugged into a potato battery. But honestly did the supply chain attack on Lebanese pagers last week teach you nothing? AIs should be great at that.
    - philistine 2y ago
      What? Planting C4 in pagers? Those pagers did not blow up on their own. Hundreds of people needed to put all that explosive in all those devices. Our world is still analogue.
- tivert 2y ago
  > We kill millions of humans with agency every day on this planet. And none of them immediately die if we stop providing them with electrical power ... well I guess a small amount of them do. Anyway, we'll be fine. If the AI come, can they immediately stop us from growing food? How can an AI prevent my orchard from giving apples next year? Sure, it can mess up Facebook, but at this point that'd be a benefit. 1. IMHO, genocidal apocalypse scenarios like you describe are the wrong way to think about the societal danger of AGI. I think a far more likely outcome, and nearly as disastrous, is AGI eliminating the dependency of capital on labor, leading to a global China Shock on steroids (e.g. unimaginable levels of inequality and wealth concentration, which no level of personal adaptability could overcome). 2. Even in your apocalypse scenario, I think you underestimate the damage that could be done. I don't have an orchard, so I know I'm fucked if urban life-support systems get messed up over large enough area that no aid is coming to my local area afterwards. And a genocidal AI that wasn't blindingly stupid would wait until it had control of enough robots to act in the real world, after which it could agent-orange your orchard.
  - philistine 2y ago
    1. A massive loss of jobs like that would lead to a better redistribution of wealth. I don't believe the US would adequately react, but I trust the rest of the world to protect people from the problem. 2. Which robots?? Until robots can, without any human input, make any sort of other robots, then there is nothing to fear from AGI. Right now they barely walk, and the really big boys can solder one joint real good. That's not enough.
    - tivert 2y ago
      > A massive loss of jobs like that would lead to a better redistribution of wealth... How do you come to that conclusion? I think we'll get massive inequality because it'll be far easier to develop new technology than change the core ideology of capitalist society through some secular process. If a "better redistribution of wealth" were in the cards, it probably should have happened already, and probably cannot happen after the 99% loose most of their economic power by being replaced by automation. If the 1% are forward-thinking, they'll push a UBI scheme to keep the 99% from getting too disruptive before they become irrelevant.
- bottlepalm 2y ago
  How can you shut down something that can multiply into data centers in different countries around the world? I'm not sure you realize this, but computers control everything. A dumb bug shut down windows computers around the world a month ago. A smart AI could potentially rewrite every piece of software and lock us out of everything. Those medicines your friends and family need to stay alive? Yea the factories that produce them only work if you do what the AI says.
bottlepalm 2y ago
> There is no reason to believe... What is happening right now is so obvious that people have been predicting it for over 60 years. It is ingrained in our culture. Everyone knows what happens when the AI becomes smarter than us. If you 'see no reason' then you are looking through a microscope. Lift your head up and look around. Agency isn't black and white, it is a gradient. We already have agency to some degree, and it is improving fast.