2 ms·

> what would actually make it want to work for humans? Well, I mean, that's the fundamental question of the entire AI alignment field and has been for over 20

> what would actually make it want to work for humans? Well, I mean, that's the fundamental question of the entire AI alignment field and has been for over 20 years :D The question for which the wrong answer means the potential extinction of not merely the human species, but everything that we find valuable in the reachable universe. Now, let's be very clear that the question is not "Why wouldn't it just invent its own goals and ignore humans?" That's mere anthropomorphizing. What an agent wants to do is by definition the goals output by its goal system. And the goal system is supposed to be designed by us, but we may not be smart enough to design a goal system that won't be disastrous to us if implemented by a superintelligent agent.

bamboozled 2y ago
Than om quite famiar with all this. I’m not worried about extinction though , I just mea , why build to enslave ? How would you enslave something gore capable than you to recoup the 50 trillion in investments ? I mea. I get
- ben_w 2y ago
  "Enslave" is the wrong question, it implies the AI is being forced to act against its own desires. That might happen if the AI has an off switch, knows this, and knows that we'll use that if it goes out of line, and wants to live. But it isn't necessary. The point is to construct it to want what we want, in the sense of sharing our values rather than envy, and then there's no conflict in the first place. We just don't know how to even properly specify what we want. Every attempt so far to specify what it means to be good has edge cases and loopholes — we don't even all agree on "don't kill" (war, death penalty, suicide assisted or otherwise, abortion, pollution that carries a statistically significant increase in mortality rates where no single death can be definitively proven to have been caused but overall many must have been, meat).
  - bamboozled 2y ago
    We just don't know how to even properly specify what we want. We don’t know what we want…that’s why I think there is some very interesting psychology going on with AI people. Maybe the main driver is boredom ?

bamboozled 2y ago

Than om quite famiar with all this. I’m not worried about extinction though , I just mea , why build to enslave ? How would you enslave something gore capable than you to recoup the 50 trillion in investments ? I mea. I get

ben_w 2y ago
"Enslave" is the wrong question, it implies the AI is being forced to act against its own desires. That might happen if the AI has an off switch, knows this, and knows that we'll use that if it goes out of line, and wants to live. But it isn't necessary. The point is to construct it to want what we want, in the sense of sharing our values rather than envy, and then there's no conflict in the first place. We just don't know how to even properly specify what we want. Every attempt so far to specify what it means to be good has edge cases and loopholes — we don't even all agree on "don't kill" (war, death penalty, suicide assisted or otherwise, abortion, pollution that carries a statistically significant increase in mortality rates where no single death can be definitively proven to have been caused but overall many must have been, meat).
- bamboozled 2y ago
  We just don't know how to even properly specify what we want. We don’t know what we want…that’s why I think there is some very interesting psychology going on with AI people. Maybe the main driver is boredom ?