4 ms·

Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical?

Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? There’s a chance that AGI will have no interest in harming humanity, too. But people talk like it’s a foregone conclusion.

JoshTriplett 4y ago
> After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? 1) If created correctly, it isn't a life form. 2) "life form" and any kind of reasoning from non-artificial entities will almost certainly lead to incorrect conclusions. 3) Destroying humanity is unethical by any non-broken value system. > There’s a chance that AGI will have no interest in harming humanity, too. There's a chance that all the air molecules in the room will all simultaneously be on the opposite side, causing someone to suffocate. But it's an vast understatement to say that that's mind-bogglingly unlikely. The most likely scenario is that AGI has no idea what "humanity" is. You don't have to be the AGI's "enemy" to be made of matter that it isn't prohibited from repurposing elsewhere. > But people talk like it’s a foregone conclusion. It's the default without substantial work to the contrary. And even if it wasn't, something doesn't have to be a foregone conclusion to be too dangerous. Nobody should be individually considering whether to blow up the sun, either.
- mxkopy 4y ago
  How are you so sure about any of this? I'm not sure we've defined humanity's interests well enough for us to say some action is for or against it. Knowing where air will go is one thing; knowing whether or not something is 'of benefit' is, I think, in a completely different realm. Especially considering an agent more intelligent than humanity as a collective. Maybe the 'interests of humanity' are undecidable, and the AGI that takes the actions that benefit them most uses an understanding completely orthogonal to ours, purely by accident. How do you know that this is less likely than not?
  - Silverback_VII 4y ago
    Why do I have the feeling that I'm reading the rationalizations of a species which about to disappear?
    - mxkopy 4y ago
      Too much Twitter? Not enough sci-fi? I suggest Philip K Dick for this condition. For every condition, tbh
- scbrg 4y ago
  > 3) Destroying humanity is unethical by any non-broken value system. Nah. Given the ridiculous amount of damage humanity is doing to its environment and other lifeforms there's a good case to be made for destroying it for the greater good.
  - dragonwriter 4y ago
    Whose greater good?
    - scbrg 4y ago
      The rest of the lifeforms. Those here on earth and those we might encounter if we manage to leave the solar system - unlikely as that may be. I'm not saying that humanity necessarily should be destroyed, I'm just saying that the statement "Destroying humanity is unethical by any non-broken value system" is simplistic. If you put any value at all on non human life, it eventually becomes a numbers game. One which I'm not certain "humanity" is necessarily winning.
  - tejohnso 4y ago
    Not sure why you're being downvoted after offering a reasonable point. I'd go even further and say that not destroying humanity would be unethical by any non human centric value system. There is little debate that we are in the midst of a mass extinction and that it is human caused. Perhaps it might be more ethical to destroy human civilization rather than humanity itself. But I think there's a good case to be made either way.
  - philwelch 4y ago
    You first.
- pixl97 4y ago
  Things like the noble gas law describe why it's improbable that a spontaneous vacuum will form, the problem here is you're playing fast and loose with scientific 'law' as an analogy. Much more complex systems built on top of systems are in use here. Nth order effects are both fun and impossible to fully predict. It is also incredibly odd to think AGI would not know what humanity is as the corpus of information that will be used trained said AGI will be the sum knowledge of humanity. The number of misguided ideas used so far begs for the dismissal of the arguments you've made.
naasking 4y ago
> Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? Is it unethical to teach your kids manners and ethics so they're civilized? That's what alignment is, not whatever you're trying to paint it as. Creating a murderous AGI is unethical, creating one with ethics is not. Not sure how you got this so backwards.

JoshTriplett 4y ago

> After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? 1) If created correctly, it isn't a life form. 2) "life form" and any kind of reasoning from non-artificial entities will almost certainly lead to incorrect conclusions. 3) Destroying humanity is unethical by any non-broken value system. > There’s a chance that AGI will have no interest in harming humanity, too. There's a chance that all the air molecules in the room will all simultaneously be on the opposite side, causing someone to suffocate. But it's an vast understatement to say that that's mind-bogglingly unlikely. The most likely scenario is that AGI has no idea what "humanity" is. You don't have to be the AGI's "enemy" to be made of matter that it isn't prohibited from repurposing elsewhere. > But people talk like it’s a foregone conclusion. It's the default without substantial work to the contrary. And even if it wasn't, something doesn't have to be a foregone conclusion to be too dangerous. Nobody should be individually considering whether to blow up the sun, either.

mxkopy 4y ago
How are you so sure about any of this? I'm not sure we've defined humanity's interests well enough for us to say some action is for or against it. Knowing where air will go is one thing; knowing whether or not something is 'of benefit' is, I think, in a completely different realm. Especially considering an agent more intelligent than humanity as a collective. Maybe the 'interests of humanity' are undecidable, and the AGI that takes the actions that benefit them most uses an understanding completely orthogonal to ours, purely by accident. How do you know that this is less likely than not?
- Silverback_VII 4y ago
  Why do I have the feeling that I'm reading the rationalizations of a species which about to disappear?
  - mxkopy 4y ago
    Too much Twitter? Not enough sci-fi? I suggest Philip K Dick for this condition. For every condition, tbh
scbrg 4y ago
> 3) Destroying humanity is unethical by any non-broken value system. Nah. Given the ridiculous amount of damage humanity is doing to its environment and other lifeforms there's a good case to be made for destroying it for the greater good.
- dragonwriter 4y ago
  Whose greater good?
  - scbrg 4y ago
    The rest of the lifeforms. Those here on earth and those we might encounter if we manage to leave the solar system - unlikely as that may be. I'm not saying that humanity necessarily should be destroyed, I'm just saying that the statement "Destroying humanity is unethical by any non-broken value system" is simplistic. If you put any value at all on non human life, it eventually becomes a numbers game. One which I'm not certain "humanity" is necessarily winning.
- tejohnso 4y ago
  Not sure why you're being downvoted after offering a reasonable point. I'd go even further and say that not destroying humanity would be unethical by any non human centric value system. There is little debate that we are in the midst of a mass extinction and that it is human caused. Perhaps it might be more ethical to destroy human civilization rather than humanity itself. But I think there's a good case to be made either way.
- philwelch 4y ago
  You first.
pixl97 4y ago
Things like the noble gas law describe why it's improbable that a spontaneous vacuum will form, the problem here is you're playing fast and loose with scientific 'law' as an analogy. Much more complex systems built on top of systems are in use here. Nth order effects are both fun and impossible to fully predict. It is also incredibly odd to think AGI would not know what humanity is as the corpus of information that will be used trained said AGI will be the sum knowledge of humanity. The number of misguided ideas used so far begs for the dismissal of the arguments you've made.

naasking 4y ago

> Part of why I like the topic is because it’s so incendiary. After all, you’re trying to create and control a new life form. Isn’t that a teensy bit unethical? Is it unethical to teach your kids manners and ethics so they're civilized? That's what alignment is, not whatever you're trying to paint it as. Creating a murderous AGI is unethical, creating one with ethics is not. Not sure how you got this so backwards.