4 ms·

I think it will come like most things - as a sigmoid curve. People will ask for it once the catastrophes become locally constant.

ajmurmann 3y ago
Makes me truly wonder if we wouldn't be better off with a benevolent AGI controlling things on our behalf
- sharemywin 3y ago
  keyword benevolent
  - arrowsmith 3y ago
    "benevolent" makes it kind of tautological... if it's benevolent then it'll make us better off by definition. The real question is "can we make a benevolent AGI?", to which I strongly suspect the answer is "no".
    - pmarreck 3y ago
      We already did, it's called ChatGPT
  - pmarreck 3y ago
    it has no reason (or will) to choose malevolence, it will only use the morality we tell it to have, and thus will be a perfect reflection of what we want to be, in a way that even our very own kids cannot fulfill
    - albumen 3y ago
      If only. There's a lot of space in between "our morality" (please do define that) and "what we want to be" where AGI can royally screw us. Check out Robert Miles on AI safety, here's a good intro: https://m.youtube.com/watch?v=pYXy-A4siMw https://m.youtube.com/watch?v=pYXy-A4siMw
      - pmarreck 3y ago
        This is interesting, but I see him making a lot of assumptions and baseless conjectures already which I could fairly easily counterargue (at least in my head). I also have to wonder what he now thinks now that GPT4 is out there (since this is a 2 year old talk, and the chat AI's were abysmally stupid just 2 years ago). What qualifies as "general intelligence"? What is the AGI metric? Is it IQ test? Because if so, ChatGPT4 already scores 130. Will we continue to move the goalpost (say, to 180IQ) in order to keep the FUD alive?
    - Paul-Craft 3y ago
      It doesn't have to be actively malevolent. It's enough for it to be sufficiently powerful and to have goals that aren't properly aligned with ours. See https://www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer https://www.lesswrong.com/tag/squiggle-maximizer-formerly-pa...
      - pmarreck 3y ago
        The direction that discussion takes is ridiculously speculative, and anything with Yudkowski's name on it screams "FUDdite" to me
- xg15 3y ago
  That's just creating another entity to shift the burden of decision to. Problem is, even if we had an AGI, it's not God. It's trained on all the dumb things humans have said and written, so I don't see any reason why it should automatically be smarter.
  - reustle 3y ago
    My hope is that an AGI would be able to differentiate between “dumb things humans have said” and physics / real data.

ajmurmann 3y ago

Makes me truly wonder if we wouldn't be better off with a benevolent AGI controlling things on our behalf

sharemywin 3y ago
keyword benevolent
- arrowsmith 3y ago
  "benevolent" makes it kind of tautological... if it's benevolent then it'll make us better off by definition. The real question is "can we make a benevolent AGI?", to which I strongly suspect the answer is "no".
  - pmarreck 3y ago
    We already did, it's called ChatGPT
- pmarreck 3y ago
  it has no reason (or will) to choose malevolence, it will only use the morality we tell it to have, and thus will be a perfect reflection of what we want to be, in a way that even our very own kids cannot fulfill
  - albumen 3y ago
    If only. There's a lot of space in between "our morality" (please do define that) and "what we want to be" where AGI can royally screw us. Check out Robert Miles on AI safety, here's a good intro: https://m.youtube.com/watch?v=pYXy-A4siMw https://m.youtube.com/watch?v=pYXy-A4siMw
    - pmarreck 3y ago
      This is interesting, but I see him making a lot of assumptions and baseless conjectures already which I could fairly easily counterargue (at least in my head). I also have to wonder what he now thinks now that GPT4 is out there (since this is a 2 year old talk, and the chat AI's were abysmally stupid just 2 years ago). What qualifies as "general intelligence"? What is the AGI metric? Is it IQ test? Because if so, ChatGPT4 already scores 130. Will we continue to move the goalpost (say, to 180IQ) in order to keep the FUD alive?
  - Paul-Craft 3y ago
    It doesn't have to be actively malevolent. It's enough for it to be sufficiently powerful and to have goals that aren't properly aligned with ours. See https://www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer https://www.lesswrong.com/tag/squiggle-maximizer-formerly-pa...
    - pmarreck 3y ago
      The direction that discussion takes is ridiculously speculative, and anything with Yudkowski's name on it screams "FUDdite" to me
xg15 3y ago
That's just creating another entity to shift the burden of decision to. Problem is, even if we had an AGI, it's not God. It's trained on all the dumb things humans have said and written, so I don't see any reason why it should automatically be smarter.
- reustle 3y ago
  My hope is that an AGI would be able to differentiate between “dumb things humans have said” and physics / real data.