2 ms·

Those four assumptions have nothing to do with danger and everything to do with capability. I think a more interesting question is "how do we make an intellige

by cstross 11y ago

Those four assumptions have nothing to do with danger and everything to do with capability. I think a more interesting question is "how do we make an intelligence, artificial or human, that isn't potentially dangerous?" Humans have a bunch of low-level primate aggression that seems to be wired in; we can modulate it (we're largely self-domesticating) but we can't totally get rid of it and more importantly, we can't look at a baby's genome (and nurturing environment) and say "yup, this one's going to grow up to be Hitler", or "this 'un's going to be a pacifist, altruist, and general benefactor". On the other hand we can probably add a bunch of monitoring code to any AI we develop rather than grow: if nothing else, consider the possibility of simulation runs looking for an empathy deficiency.

jamii 11y ago
You're right that the interesting question is how to make a friendly intelligence, but all the groups who are trying to draw attention to that problem are running into the wall of doubt about capability. The OP said: > I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human. Since an actual human might be out to get me already the question is not motives but capability - should I fear evil AIs more than I fear evil humans? Or an evil human with a friendly leashed AI? The argument typically revolves around whether or not AI would self-improve and lead to runaway intelligence, but I think that that's a distraction - there are plenty of other ways AI could be dangerous that are easier to persuade people with without running into the that-sounds-like-scifi filter.
jamii 11y ago
Also, to a first approximation capability == danger. Just look at what even well-meaning humans have managed to do - imagine an AI that could eg design memes more persuasive than Marxism. We can't even get humans to agree on ethical values, let alone explain them to an AI, so even a perfectly altruistic AI could still be a disaster if it has the wrong value system, or if we don't perfectly understand the consequences of our own values systems when fully enacted.

by cstross 11y ago

jamii 11y ago

You're right that the interesting question is how to make a friendly intelligence, but all the groups who are trying to draw attention to that problem are running into the wall of doubt about capability. The OP said: > I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human. Since an actual human might be out to get me already the question is not motives but capability - should I fear evil AIs more than I fear evil humans? Or an evil human with a friendly leashed AI? The argument typically revolves around whether or not AI would self-improve and lead to runaway intelligence, but I think that that's a distraction - there are plenty of other ways AI could be dangerous that are easier to persuade people with without running into the that-sounds-like-scifi filter.

jamii 11y ago

Also, to a first approximation capability == danger. Just look at what even well-meaning humans have managed to do - imagine an AI that could eg design memes more persuasive than Marxism. We can't even get humans to agree on ethical values, let alone explain them to an AI, so even a perfectly altruistic AI could still be a disaster if it has the wrong value system, or if we don't perfectly understand the consequences of our own values systems when fully enacted.