5 ms·

A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sa

A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sandbox? Do they care what happens to any ants or other bugs that might live in it? There does not need to be a "how", as you put it. The logic is "Maybe we should tread carefully when creating an intelligence that is magnitudes beyond our own". The logic is "Maybe we should tread carefully with these technologies considering they have already progressed to the point where the creators go 'We're not sure what's happening inside the box and it's also doing things we didn't think it could do'". To just go barreling forward because "Hurr durr that's just nonsense!" is the height of ignorance and not something I expect from this forum.

entropyneur 4y ago
Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.
- hamburga 4y ago
  https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
- munificent 4y ago
  > Most bugs haven't gone extinct though. We're working on it as fast as we can: https://en.wikipedia.org/wiki/Decline_in_insect_populations https://en.wikipedia.org/wiki/Decline_in_insect_populations https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752634/the-insect-crisis-oliver-milman https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752... https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/ https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-A...
computerex 4y ago
Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.
- joenot443 4y ago
  There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.
  - echelon 4y ago
    Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddle with the power grid yet?
    - PaulDavisThe1st 4y ago
      > Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.
      - echelon 4y ago
        And is it there yet? Does ChatGPT have its finger on the trigger? I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers. The response to "AI will turn the world to paperclips" is "LOL" The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpredictably" is "yeah, we should be careful"
        PaulDavisThe1st 4y ago
        Of course it's not there yet. For once (?) we are having this discussion before the wolves are at the door. And yes, there are more important things to worry about right now than the AIpocalypse. But that doesn't mean that thinking about what happens as (some) humans come to trust and rely on these systems isn't important.
bamboozled 4y ago
If such a thing as a super intelligence existed, why would it operate on matter in the physical world? Animals do physical world modifications because we're biological and need shelter etc. A super intelligence would quickly understand everything there is to know about the physical world and just move on meta-physics. It would just be like an "orb". Humans are already bored with the physical world and thus much prefer being in virtual spaces, look at everyone just starting at Instagram. IMO This idea that it's going to eat all the atoms on Earth is part of the anthropomorphize of something mythological. Exactly how we imagine God as a dude with a grey beard.

entropyneur 4y ago

Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.

hamburga 4y ago
https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence
munificent 4y ago
> Most bugs haven't gone extinct though. We're working on it as fast as we can: https://en.wikipedia.org/wiki/Decline_in_insect_populations https://en.wikipedia.org/wiki/Decline_in_insect_populations https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752634/the-insect-crisis-oliver-milman https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752... https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-APOCALYPSE/egpbykdxjvq/ https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-A...

computerex 4y ago

Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.

joenot443 4y ago
There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.
- echelon 4y ago
  Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddle with the power grid yet?
  - PaulDavisThe1st 4y ago
    > Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.
    - echelon 4y ago
      And is it there yet? Does ChatGPT have its finger on the trigger? I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers. The response to "AI will turn the world to paperclips" is "LOL" The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpredictably" is "yeah, we should be careful"
      - PaulDavisThe1st 4y ago
        Of course it's not there yet. For once (?) we are having this discussion before the wolves are at the door. And yes, there are more important things to worry about right now than the AIpocalypse. But that doesn't mean that thinking about what happens as (some) humans come to trust and rely on these systems isn't important.

bamboozled 4y ago

If such a thing as a super intelligence existed, why would it operate on matter in the physical world? Animals do physical world modifications because we're biological and need shelter etc. A super intelligence would quickly understand everything there is to know about the physical world and just move on meta-physics. It would just be like an "orb". Humans are already bored with the physical world and thus much prefer being in virtual spaces, look at everyone just starting at Instagram. IMO This idea that it's going to eat all the atoms on Earth is part of the anthropomorphize of something mythological. Exactly how we imagine God as a dude with a grey beard.