4 ms·

Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to m

by HKH2 3y ago

Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?

kouru225 3y ago
What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
- HKH2 3y ago
  > What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
  - kouru225 3y ago
    “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
    - HKH2 3y ago
      > And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
      - kouru225 3y ago
        >Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) >Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me. Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
        HKH2 3y ago
        I have the belief that humans are innately selfish and shortsighted. Obviously corruption is not viable long-term. Laws should make corruption not strategically beneficial, but if you can find loopholes or people to bribe, they are more or less irrelevant. What would make an AGI not try to game the system?
        kouru225 3y ago
        Lemme ask you: if murder were legal do you think we’d be able to build and produce so much? Of course not. It doesn’t matter what the laws are. Reality makes corruption strategically unviable.
        HKH2 3y ago
        True. Let me be more precise: the legal system forces (or is meant to force) shortsighted people to deal with long-term consequences. The law is constantly revised to preempt future consequences, but it is not perfect. (Ex post facto laws are used to punish people who have committed actions that were legal at the time. Loopholes will always exist because language is always somewhat imprecise.) Now, you might say that AGI will be able to perfectly predict the future of all actions and make the legal system redundant. Ignoring the absurd amount of possibilities, I struggle to believe it's possible because an AGI has no sense of pain to give actions weight; it just has to follow human precedent. Seeming good enough, it could steer humanity into a dead end unknowingly. Also, it might decide that a few eggs have to be broken to make a utopian omelette.

by HKH2 3y ago

Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?

kouru225 3y ago

What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.

HKH2 3y ago
> What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
- kouru225 3y ago
  “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
  - HKH2 3y ago
    > And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
    - kouru225 3y ago
      >Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) >Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me. Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
      - HKH2 3y ago
        I have the belief that humans are innately selfish and shortsighted. Obviously corruption is not viable long-term. Laws should make corruption not strategically beneficial, but if you can find loopholes or people to bribe, they are more or less irrelevant. What would make an AGI not try to game the system?
        kouru225 3y ago
        Lemme ask you: if murder were legal do you think we’d be able to build and produce so much? Of course not. It doesn’t matter what the laws are. Reality makes corruption strategically unviable.
        HKH2 3y ago
        True. Let me be more precise: the legal system forces (or is meant to force) shortsighted people to deal with long-term consequences. The law is constantly revised to preempt future consequences, but it is not perfect. (Ex post facto laws are used to punish people who have committed actions that were legal at the time. Loopholes will always exist because language is always somewhat imprecise.) Now, you might say that AGI will be able to perfectly predict the future of all actions and make the legal system redundant. Ignoring the absurd amount of possibilities, I struggle to believe it's possible because an AGI has no sense of pain to give actions weight; it just has to follow human precedent. Seeming good enough, it could steer humanity into a dead end unknowingly. Also, it might decide that a few eggs have to be broken to make a utopian omelette.