4 ms·

I think those 2 videos are on the topic: - AI "Stop Button" Problem https://www.youtube.com/watch?v=3TYT1QfdfsM https://www.youtube.com/watch?v=3TYT1QfdfsM (th

I think those 2 videos are on the topic: - AI "Stop Button" Problem https://www.youtube.com/watch?v=3TYT1QfdfsM https://www.youtube.com/watch?v=3TYT1QfdfsM (the AI won't let you push the stop button, because that would impair its function) - General AI Won't Want You To Fix its Code https://www.youtube.com/watch?v=4l7Is6vOAOA https://www.youtube.com/watch?v=4l7Is6vOAOA (the AI won't let you patch, because changing its goal means it can't reach it)

grondilu 10y ago
This second video is of particular interest to me, as it tackles a problem I've been thinking quite a lot lately : is self-modification compatible with volition? What's the point of wanting something if you can chose what to want? The pill that makes you love the idea of killing your kids is an extreme example, and probably no-one would accept to take this pill, but it's really not so obvious why. Or at least, the reasons are pretty deep. BTW IMHO a simpler example would be a meta-suicide pill : instead of being a poison that would just kill you, it's a pill that would make you want to kill yourself. I suppose you can make people want to kill themselves by inflicting them an awful chronic pain such that death would look like the only solution, but the pill I'm thinking about would not do that : it would change your mind so that you would want to die just because you like the idea of dying, for no particular reason.
- maxerickson 10y ago
  Are you implicitly arguing that humans going about their day to day lives have a self that they are incapable of modifying? What's the difference between a pill and an idea?
- algorithmsRcool 10y ago
  If this is an interesting topic, give "Permutation City" by "Greg Egan" a read. It involves a digitized character that intentionally modifies itself in order to avoid unhappiness.
  - chaosmachine 10y ago
    Permutation City is actually by Greg Egan.
    - algorithmsRcool 10y ago
      Ah, thank you. I read Permutation City in January right after I read Neuromancer and apparently got my wires crossed.
      - jacobush 10y ago
        Wires permutated
- garyrob 10y ago
  "is self-modification compatible with volition?" That's a plot point of WestWorld. A more interesting TV series than I expected.
vl 10y ago
In addition to these, Nate Soares: "Ensuring Smarter-than-Human Intelligence has a Positive Outcome" https://youtu.be/dY3zDvoLoao https://youtu.be/dY3zDvoLoao talks about how current AI fears misrepresent the problem and gives multiple examples of difficulty of defining correct objective function. I enjoyed this explanation a lot!

grondilu 10y ago

This second video is of particular interest to me, as it tackles a problem I've been thinking quite a lot lately : is self-modification compatible with volition? What's the point of wanting something if you can chose what to want? The pill that makes you love the idea of killing your kids is an extreme example, and probably no-one would accept to take this pill, but it's really not so obvious why. Or at least, the reasons are pretty deep. BTW IMHO a simpler example would be a meta-suicide pill : instead of being a poison that would just kill you, it's a pill that would make you want to kill yourself. I suppose you can make people want to kill themselves by inflicting them an awful chronic pain such that death would look like the only solution, but the pill I'm thinking about would not do that : it would change your mind so that you would want to die just because you like the idea of dying, for no particular reason.

maxerickson 10y ago
Are you implicitly arguing that humans going about their day to day lives have a self that they are incapable of modifying? What's the difference between a pill and an idea?
algorithmsRcool 10y ago
If this is an interesting topic, give "Permutation City" by "Greg Egan" a read. It involves a digitized character that intentionally modifies itself in order to avoid unhappiness.
- chaosmachine 10y ago
  Permutation City is actually by Greg Egan.
  - algorithmsRcool 10y ago
    Ah, thank you. I read Permutation City in January right after I read Neuromancer and apparently got my wires crossed.
    - jacobush 10y ago
      Wires permutated
garyrob 10y ago
"is self-modification compatible with volition?" That's a plot point of WestWorld. A more interesting TV series than I expected.

vl 10y ago

In addition to these, Nate Soares: "Ensuring Smarter-than-Human Intelligence has a Positive Outcome" https://youtu.be/dY3zDvoLoao https://youtu.be/dY3zDvoLoao talks about how current AI fears misrepresent the problem and gives multiple examples of difficulty of defining correct objective function. I enjoyed this explanation a lot!