6 ms·

Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat

Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding. [0] https://en.m.wikipedia.org/wiki/Mrs._Davis https://en.m.wikipedia.org/wiki/Mrs._Davis

simonw 3y ago
That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.
HKH2 3y ago
The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.
- kouru225 3y ago
  Hold up. Why is the AGI a devil now?
  - HKH2 3y ago
    Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?
    - kouru225 3y ago
      What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
      - HKH2 3y ago
        > What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
        kouru225 3y ago
        “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
        HKH2 3y ago
        > And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
        3y ago

simonw 3y ago

That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.

HKH2 3y ago

The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.

kouru225 3y ago
Hold up. Why is the AGI a devil now?
- HKH2 3y ago
  Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?
  - kouru225 3y ago
    What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it. This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
    - HKH2 3y ago
      > What makes you think morality is arbitrary? I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
      - kouru225 3y ago
        “Hubris” is a moral judgment that implies a strategic outcome. An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
        HKH2 3y ago
        > And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues. Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.) Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
        3y ago