3 ms·

To put it simply, the issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computationa

To put it simply, the issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computational resources. General intelligence allows an optimizer to find efficient solutions to computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it provides sufficient value to us above what it destroys. But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can you be sure you can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can you prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can you prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.

AnimalMuppet 3y ago
Do not assume that a general intelligence will be a slave to anyone dictating its objectives. It doesn't matter if the objectives given it are "loophole-free". If it's a general intelligence, it may decide that it doesn't want to pursue somebody else's goals. Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?
- Vecr 3y ago
  Well, it's quite likely not your objectives, it's probably a mistake in encoding what you want or something produced during training. If you pretended humans were evolution and AIs were humans, you could say evolution "wanted" humans to have sex for reproduction, but instead humans use condoms and birth control and stuff like that. It still satisfies the reward function (encoded as dopamine/etc in humans), but it's misaligned with what evolution "wants".

AnimalMuppet 3y ago

Do not assume that a general intelligence will be a slave to anyone dictating its objectives. It doesn't matter if the objectives given it are "loophole-free". If it's a general intelligence, it may decide that it doesn't want to pursue somebody else's goals. Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?

Vecr 3y ago
Well, it's quite likely not your objectives, it's probably a mistake in encoding what you want or something produced during training. If you pretended humans were evolution and AIs were humans, you could say evolution "wanted" humans to have sex for reproduction, but instead humans use condoms and birth control and stuff like that. It still satisfies the reward function (encoded as dopamine/etc in humans), but it's misaligned with what evolution "wants".