3 ms·

>If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal p

>If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about… I wish this would stop being repeated. The argument for a solar system tiling smiley face machine has been thoroughly refuted.

DanAndersen 11y ago
Would you be able to provide links to the arguments that refute it? Simply asserting that someone has disproved X isn't very helpful to the rest of us.
- loup-vaillant 11y ago
  Seconded. I have seen a lot of smiley examples, and no serious refutation yet. A link would be very appreciated.
  - jjaredsimpson 11y ago
    http://ieet.org/index.php/IEET/more/loosemore20140724 http://ieet.org/index.php/IEET/more/loosemore20140724
    - loup-vaillant 11y ago
      Thanks. I have read it all. I don't believe it. First, the bulk of his argument hinges on the conflation of optimization power and value assessment. Those are easily separated. If you program an AI with a given utility function, and a goal of maximising that utility function, it won't modify the utility function in the course of increasing its optimization power. The utility function is supposed to be an invariant. Now sure, you could try and tell the AI to align its own utility function to that of the human race… whatever that means. What is a human? How do you resolve conflicting values? It is all well and good to assume the presence of checking code, but he gave no hint about how those checks might actually work. If those are mere human-crafted checks, they are bound to be circumvented eventually, as the AI increases its optimization power beyond human level. The AI doesn't care about human values. It only cares about maximising whatever goal it has been provided with, within the constraints of the checking code. Second, I am wary of those swarm constraint relaxation intelligences. Since it is not even clear what utility function they might be maximising (if any), I wonder what would happen if they are given the power to self-modify in any significant way. Sure, this might be a lead worth following, but the safety of this design is even murkier. Third, I'm not sure the guys at MIRI even believe in the doctrine of logical infallibility. They are well aware that perfect Bayesian reasoning requires super-exponential computational power, and is thus impossible to perform exactly. The AI is bound to approximate the Bayesian ideal, and cannot assume its reasoning is perfect —except in the few cases where it managed to produce an actual actual proof, like in 1+1=2, or simple probability calculations with simple prior information. Overall, all I see is weak arguments grounded in assumptions I don't share. Worth a read, though.

DanAndersen 11y ago

Would you be able to provide links to the arguments that refute it? Simply asserting that someone has disproved X isn't very helpful to the rest of us.

loup-vaillant 11y ago
Seconded. I have seen a lot of smiley examples, and no serious refutation yet. A link would be very appreciated.
- jjaredsimpson 11y ago
  http://ieet.org/index.php/IEET/more/loosemore20140724 http://ieet.org/index.php/IEET/more/loosemore20140724
  - loup-vaillant 11y ago
    Thanks. I have read it all. I don't believe it. First, the bulk of his argument hinges on the conflation of optimization power and value assessment. Those are easily separated. If you program an AI with a given utility function, and a goal of maximising that utility function, it won't modify the utility function in the course of increasing its optimization power. The utility function is supposed to be an invariant. Now sure, you could try and tell the AI to align its own utility function to that of the human race… whatever that means. What is a human? How do you resolve conflicting values? It is all well and good to assume the presence of checking code, but he gave no hint about how those checks might actually work. If those are mere human-crafted checks, they are bound to be circumvented eventually, as the AI increases its optimization power beyond human level. The AI doesn't care about human values. It only cares about maximising whatever goal it has been provided with, within the constraints of the checking code. Second, I am wary of those swarm constraint relaxation intelligences. Since it is not even clear what utility function they might be maximising (if any), I wonder what would happen if they are given the power to self-modify in any significant way. Sure, this might be a lead worth following, but the safety of this design is even murkier. Third, I'm not sure the guys at MIRI even believe in the doctrine of logical infallibility. They are well aware that perfect Bayesian reasoning requires super-exponential computational power, and is thus impossible to perform exactly. The AI is bound to approximate the Bayesian ideal, and cannot assume its reasoning is perfect —except in the few cases where it managed to produce an actual actual proof, like in 1+1=2, or simple probability calculations with simple prior information. Overall, all I see is weak arguments grounded in assumptions I don't share. Worth a read, though.