4 ms·

Okay, but let’s be pessimistic for a moment. What can we do if that revolution does happen, and they’re close to AGI? I don’t believe the control problem is so

Okay, but let’s be pessimistic for a moment. What can we do if that revolution does happen, and they’re close to AGI? I don’t believe the control problem is solved, but I’m not sure it would matter if it is.

ForHackernews 2y ago
Being pessimistic, how come no human supergeniuses ever took over the world? Why didn't Leibniz make everyone else into his slaves? I don't even understand what the proposed mechanism for "rouge AI enslaves humanity" is. It's scifi (and not hard scifi) as far as I can see.
- Philpax 2y ago
  Once you have one AGI, you can scale it to many AGI as long as you have the necessary compute. An AGI never needs to take breaks, can work non-stop on a problem, has access to all of the world's information simultaneously, and can interact with any system it's connected to. To put it simply, it could outcompete humanity on every metric that matters, especially given recent advancements in robotics.
  - ForHackernews 2y ago
    ...so it can think really hard all the time and come up with lots of great, devious evil ideas? Again, I wonder why no group of smart people with brilliant ideas has unilaterally imposed those ideas on the rest of humanity through sheer force of genius.
    - Philpax 2y ago
      An equivalent advance in autonomous robotics would solve the force projection issue, if that's what you're getting at. I don't know if this will happen with any certainty, but the general idea of commoditising intelligence very much has the ability to tip the world order: every problem that can be tackled by throwing brainpower at it will be, and those advances will compound. Also, the question you're posing did happen: it was called the Manhattan Project.
      - redserk 2y ago
        And if this whole exercise turns out to be a flop and gets us absolutely nowhere closer to AGI? “AGI” has proven to be today’s hot marketing stunt for when you need to raise another round of cash and your only viable product is optimism. Flying cars were just around the corner in the 60s, too.
        anon84873628 2y ago
        This thread started from a deliberately pessimistic hypothetical of what happens if AGI actually manifests, so your comment is misplaced.
        arisAlexis 2y ago
        You really haven't used any LLM seriously eh
      - ForHackernews 2y ago
        So don't plug the smart evil computer into the strong robots? Great, AI apocalypse averted. The Manhattan Project would be a cute example if the Los Alamos scientists had gone rogue and declared themselves emperors of mankind, but no, in fact the people in charge remained the people in charge - mostly not supergeniuses.
    - lupire 2y ago
      Look at any corporation or government to understand how a large group of humans can be driven to do specific things none of them individually want.
    - jprete 2y ago
      Quite a few have succeeded in conquering large fractions of the Earth's population: Napoleon, Hitler, Genghis Khan, the Roman emperors, Alexander the Great, Mao Zedong. America and Britain as systems did so for long periods of time. All of these entities would have been enormously more powerful with access to an AGI's immortality, sleeplessness, and ability to clone itself.
      - SketchySeaBeast 2y ago
        I can see what you're trying to say, but I cannot for the life of me figure out how an AGI would have helped Alexander the Great.
        jprete 2y ago
        Alexander the Great made his conquests by building a really good reputation for war, then leveraging it to get tribute agreements while leaving the local governments intact. This is a good way to do it when communication lines are slow and unreliable, because the emperor just needs to check tribute once a year to enforce the agreements, but it's weak control. If Alexander could have left perfectly aligned copies of himself in every city he passed, he could have gotten much more control and authority, and still avoided a fight by agreeing to maintain the local power structure with himself as the new head of state.
        SketchySeaBeast 2y ago
        Oh, you're assuming an entire networking infrastructure as well. That makes way more sense, but the miracle there isn't AGI - without networking they'd lose alignment over time. Honestly, I feel like it would devolve in a patchwork of different kingdoms run by an Alexander figurehead... where have I seen this before? The problem you're proposing could be solved via a high quality cellular network.
      - anon84873628 2y ago
        And of course the more society is wired up and controlled by computer systems, the more the AGI could directly manage it.
- z3phyr 2y ago
  I consider many successful military leaders and politicians to be geniuses as well. In my books, Caesar is as genius as Newton! Having said that, we do not to understand the world to exploit it for ourselves. And what better way to understand and exploit the universe than science? Its an endearment.
- HeatrayEnjoyer 2y ago
  > Being pessimistic, how come no human supergeniuses ever took over the world? Why didn't Leibniz make everyone else into his slaves? We already did. Look at the state of animals today vs <1 mya. Bovines grown in unprecedented mass numbers to live short lives before slaughter. Wolves bred into an all new animal, friendly and helpful to the dominate species. Previously apex predators with claws, teeth, speed and strength, rendered extinct.
  - adalacelove 2y ago
    Sometimes I wonder if we are going to be the unkillable plague that takes over the universe. Or maybe we will dissappear in a blink. It's hard to know, we don't have any reference point except ourselves.
    - lupire 2y ago
      Destroying human life in Earth (the only habitable place in the solar system) is far far easier than reaching something outside the solar system.
- arisAlexis 2y ago
  This is profoundly and disturbingly bad argument. 1)Leibniz wasn't superhuman 2) Leibniz couldn't work 24/7 3) he could not self increase the speed of his own hardware (body) 4) he could not spawn 1 trillion copies of him to work 24/7 Like how much time did you think before writing this
  - ForHackernews 2y ago
    Again, my reaction is... so what? A trillion hyperintelligent demons might be cogitating right now on the head of a pin. You can't prove they aren't thinking up all sorts of genius evil schemes. My point is that "intelligence" has never been a sufficient - or even necessary - component of imposing ones will on humans. I feel like HN/EA/"Grey Tribe" people fail to see this because they so worship intellect. I'm much more likely to fall victim to a big dumb man than smart computers.
    - arisAlexis 2y ago
      huh what ? this is a whole new level of flat earther thinking. you actually believe that apex predators are not the most intelligent? Like humans became the apex because of something else than intelligence? AI and covid showed humanity what levels of wacky stuff people believe. I am not trying to convince you, thank you for showing me this perspective :)

ForHackernews 2y ago

Being pessimistic, how come no human supergeniuses ever took over the world? Why didn't Leibniz make everyone else into his slaves? I don't even understand what the proposed mechanism for "rouge AI enslaves humanity" is. It's scifi (and not hard scifi) as far as I can see.

Philpax 2y ago
Once you have one AGI, you can scale it to many AGI as long as you have the necessary compute. An AGI never needs to take breaks, can work non-stop on a problem, has access to all of the world's information simultaneously, and can interact with any system it's connected to. To put it simply, it could outcompete humanity on every metric that matters, especially given recent advancements in robotics.
- ForHackernews 2y ago
  ...so it can think really hard all the time and come up with lots of great, devious evil ideas? Again, I wonder why no group of smart people with brilliant ideas has unilaterally imposed those ideas on the rest of humanity through sheer force of genius.
  - Philpax 2y ago
    An equivalent advance in autonomous robotics would solve the force projection issue, if that's what you're getting at. I don't know if this will happen with any certainty, but the general idea of commoditising intelligence very much has the ability to tip the world order: every problem that can be tackled by throwing brainpower at it will be, and those advances will compound. Also, the question you're posing did happen: it was called the Manhattan Project.
    - redserk 2y ago
      And if this whole exercise turns out to be a flop and gets us absolutely nowhere closer to AGI? “AGI” has proven to be today’s hot marketing stunt for when you need to raise another round of cash and your only viable product is optimism. Flying cars were just around the corner in the 60s, too.
      - anon84873628 2y ago
        This thread started from a deliberately pessimistic hypothetical of what happens if AGI actually manifests, so your comment is misplaced.
      - arisAlexis 2y ago
        You really haven't used any LLM seriously eh
    - ForHackernews 2y ago
      So don't plug the smart evil computer into the strong robots? Great, AI apocalypse averted. The Manhattan Project would be a cute example if the Los Alamos scientists had gone rogue and declared themselves emperors of mankind, but no, in fact the people in charge remained the people in charge - mostly not supergeniuses.
  - lupire 2y ago
    Look at any corporation or government to understand how a large group of humans can be driven to do specific things none of them individually want.
  - jprete 2y ago
    Quite a few have succeeded in conquering large fractions of the Earth's population: Napoleon, Hitler, Genghis Khan, the Roman emperors, Alexander the Great, Mao Zedong. America and Britain as systems did so for long periods of time. All of these entities would have been enormously more powerful with access to an AGI's immortality, sleeplessness, and ability to clone itself.
    - SketchySeaBeast 2y ago
      I can see what you're trying to say, but I cannot for the life of me figure out how an AGI would have helped Alexander the Great.
      - jprete 2y ago
        Alexander the Great made his conquests by building a really good reputation for war, then leveraging it to get tribute agreements while leaving the local governments intact. This is a good way to do it when communication lines are slow and unreliable, because the emperor just needs to check tribute once a year to enforce the agreements, but it's weak control. If Alexander could have left perfectly aligned copies of himself in every city he passed, he could have gotten much more control and authority, and still avoided a fight by agreeing to maintain the local power structure with himself as the new head of state.
        SketchySeaBeast 2y ago
        Oh, you're assuming an entire networking infrastructure as well. That makes way more sense, but the miracle there isn't AGI - without networking they'd lose alignment over time. Honestly, I feel like it would devolve in a patchwork of different kingdoms run by an Alexander figurehead... where have I seen this before? The problem you're proposing could be solved via a high quality cellular network.
    - anon84873628 2y ago
      And of course the more society is wired up and controlled by computer systems, the more the AGI could directly manage it.
z3phyr 2y ago
I consider many successful military leaders and politicians to be geniuses as well. In my books, Caesar is as genius as Newton! Having said that, we do not to understand the world to exploit it for ourselves. And what better way to understand and exploit the universe than science? Its an endearment.
HeatrayEnjoyer 2y ago
> Being pessimistic, how come no human supergeniuses ever took over the world? Why didn't Leibniz make everyone else into his slaves? We already did. Look at the state of animals today vs <1 mya. Bovines grown in unprecedented mass numbers to live short lives before slaughter. Wolves bred into an all new animal, friendly and helpful to the dominate species. Previously apex predators with claws, teeth, speed and strength, rendered extinct.
- adalacelove 2y ago
  Sometimes I wonder if we are going to be the unkillable plague that takes over the universe. Or maybe we will dissappear in a blink. It's hard to know, we don't have any reference point except ourselves.
  - lupire 2y ago
    Destroying human life in Earth (the only habitable place in the solar system) is far far easier than reaching something outside the solar system.
arisAlexis 2y ago
This is profoundly and disturbingly bad argument. 1)Leibniz wasn't superhuman 2) Leibniz couldn't work 24/7 3) he could not self increase the speed of his own hardware (body) 4) he could not spawn 1 trillion copies of him to work 24/7 Like how much time did you think before writing this
- ForHackernews 2y ago
  Again, my reaction is... so what? A trillion hyperintelligent demons might be cogitating right now on the head of a pin. You can't prove they aren't thinking up all sorts of genius evil schemes. My point is that "intelligence" has never been a sufficient - or even necessary - component of imposing ones will on humans. I feel like HN/EA/"Grey Tribe" people fail to see this because they so worship intellect. I'm much more likely to fall victim to a big dumb man than smart computers.
  - arisAlexis 2y ago
    huh what ? this is a whole new level of flat earther thinking. you actually believe that apex predators are not the most intelligent? Like humans became the apex because of something else than intelligence? AI and covid showed humanity what levels of wacky stuff people believe. I am not trying to convince you, thank you for showing me this perspective :)