5 ms·

this is where comes the good guy with a gun. :) There are so many resources on 'good' side that who wins is obvious.

nullsense 3y ago
It's not obvious. At all. That's like saying "people don't want to die from a novel coronavirus so it's obvious they'll take the vaccine". It feels obvious on a surface level, but it turns out reality is way more messy and unpredictable when you don't just run the caricature of it in your head but when it actually plays out for real. Nothing about how this is going to go is obvious.
- two_in_one 3y ago
  Well, at least it's obvious that humans alone will have hard time fighting superhuman AGI. BTW, it can literally fall from the sky at any moment. There enthusiasts who broadcast our location, with resources and technical level estimates. Sort of naive cargo cult, they probably think biological or artificial aliens will come with truckloads of reparation money.
gmerc 3y ago
Replace guns with nuclear weapons and you see how ridiculous the whole good you with gun excuse really is
- two_in_one 3y ago
  It's not a mass destruction weapon. But that's not the point. You have to have good guy here. No other options.
  - TeMPOraL 3y ago
    The entire thing with worrying about GAI starts with observation that it is a WMD, so powerful we can barely conceptualize it. A smart enough AGI, unless it's perfectly aligned, will end humanity (or worse - there are worse things than death), most likely by accident or just plain not caring. But it doesn't stop there. If that AI is self-improving, it could easily turn into a threat for the entire galaxy, or even the universe, unless it meets a stronger and better aligned (to its creators) alien GAI... That's the alignment 101. But I worry people don't talk about alignment 102: a perfectly aligned superhuman GAI will not destroy us, but its very existence will turn us into NPCs in the story of the future of the universe.
    - malaya_zemlya 3y ago
      The fact that we still exist speaks strongly against an AGI destroying the entire galaxy. Fermi Paradox, otoh, suggest that planetwide destruction is not out of the question.
      - digging 3y ago
        Unless we're earlier than we think. Someone has to be first.
  - digging 3y ago
    As in all systems, it is a Bad Idea to begin a process that has extremely negative possible outcomes and hope that someone will do the right thing to prevent those outcomes.

It's not obvious. At all. That's like saying "people don't want to die from a novel coronavirus so it's obvious they'll take the vaccine". It feels obvious on a surface level, but it turns out reality is way more messy and unpredictable when you don't just run the caricature of it in your head but when it actually plays out for real. Nothing about how this is going to go is obvious.

two_in_one 3y ago
Well, at least it's obvious that humans alone will have hard time fighting superhuman AGI. BTW, it can literally fall from the sky at any moment. There enthusiasts who broadcast our location, with resources and technical level estimates. Sort of naive cargo cult, they probably think biological or artificial aliens will come with truckloads of reparation money.

gmerc 3y ago

Replace guns with nuclear weapons and you see how ridiculous the whole good you with gun excuse really is

two_in_one 3y ago
It's not a mass destruction weapon. But that's not the point. You have to have good guy here. No other options.
- TeMPOraL 3y ago
  The entire thing with worrying about GAI starts with observation that it is a WMD, so powerful we can barely conceptualize it. A smart enough AGI, unless it's perfectly aligned, will end humanity (or worse - there are worse things than death), most likely by accident or just plain not caring. But it doesn't stop there. If that AI is self-improving, it could easily turn into a threat for the entire galaxy, or even the universe, unless it meets a stronger and better aligned (to its creators) alien GAI... That's the alignment 101. But I worry people don't talk about alignment 102: a perfectly aligned superhuman GAI will not destroy us, but its very existence will turn us into NPCs in the story of the future of the universe.
  - malaya_zemlya 3y ago
    The fact that we still exist speaks strongly against an AGI destroying the entire galaxy. Fermi Paradox, otoh, suggest that planetwide destruction is not out of the question.
    - digging 3y ago
      Unless we're earlier than we think. Someone has to be first.
- digging 3y ago
  As in all systems, it is a Bad Idea to begin a process that has extremely negative possible outcomes and hope that someone will do the right thing to prevent those outcomes.