4 ms·

There are a lot of assumptions you have to make to get to the point where it doesn't have the capability to be dangerous: * No improvement beyond human level (

by jamii 11y ago

There are a lot of assumptions you have to make to get to the point where it doesn't have the capability to be dangerous: * No improvement beyond human level (correlation between IQ and job-performance is variously estimated at around 0.5 and IQ largely comes down to executive function and working memory - so we have to assume an AI design that doesn't allow adding more working memory for some reason) * No extra individual capabilities (eg human-level but with mental access to Google Scholar) * No extra group capabilities (eg human-level but with telepathy between AIs - look at the difference the internet is making to human productivity eg Linux) * No horizontal scaling over time (eg one human-level AI now, one million once Moores law kicks in - that's a lot of scientists/strategists/politicians/advertisers) So in the situation where we make an AI that is somehow structurally limited to human-level intelligence and doesn't benefit from extra interfaces and we somehow can't afford to ever make many of them, then, sure, I can't think of a way that it could be dangerous. In a few minutes of cursory thought I came up with four assumptions that were necessary for assuming that they won't be more capable than humans. What are the odds that all of those hold? [EDIT isn't dangerous -> doesn't have the capability to be dangerous]

cstross 11y ago
Those four assumptions have nothing to do with danger and everything to do with capability. I think a more interesting question is "how do we make an intelligence, artificial or human, that isn't potentially dangerous?" Humans have a bunch of low-level primate aggression that seems to be wired in; we can modulate it (we're largely self-domesticating) but we can't totally get rid of it and more importantly, we can't look at a baby's genome (and nurturing environment) and say "yup, this one's going to grow up to be Hitler", or "this 'un's going to be a pacifist, altruist, and general benefactor". On the other hand we can probably add a bunch of monitoring code to any AI we develop rather than grow: if nothing else, consider the possibility of simulation runs looking for an empathy deficiency.
- jamii 11y ago
  You're right that the interesting question is how to make a friendly intelligence, but all the groups who are trying to draw attention to that problem are running into the wall of doubt about capability. The OP said: > I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human. Since an actual human might be out to get me already the question is not motives but capability - should I fear evil AIs more than I fear evil humans? Or an evil human with a friendly leashed AI? The argument typically revolves around whether or not AI would self-improve and lead to runaway intelligence, but I think that that's a distraction - there are plenty of other ways AI could be dangerous that are easier to persuade people with without running into the that-sounds-like-scifi filter.
- jamii 11y ago
  Also, to a first approximation capability == danger. Just look at what even well-meaning humans have managed to do - imagine an AI that could eg design memes more persuasive than Marxism. We can't even get humans to agree on ethical values, let alone explain them to an AI, so even a perfectly altruistic AI could still be a disaster if it has the wrong value system, or if we don't perfectly understand the consequences of our own values systems when fully enacted.

by jamii 11y ago

cstross 11y ago

Those four assumptions have nothing to do with danger and everything to do with capability. I think a more interesting question is "how do we make an intelligence, artificial or human, that isn't potentially dangerous?" Humans have a bunch of low-level primate aggression that seems to be wired in; we can modulate it (we're largely self-domesticating) but we can't totally get rid of it and more importantly, we can't look at a baby's genome (and nurturing environment) and say "yup, this one's going to grow up to be Hitler", or "this 'un's going to be a pacifist, altruist, and general benefactor". On the other hand we can probably add a bunch of monitoring code to any AI we develop rather than grow: if nothing else, consider the possibility of simulation runs looking for an empathy deficiency.

jamii 11y ago
You're right that the interesting question is how to make a friendly intelligence, but all the groups who are trying to draw attention to that problem are running into the wall of doubt about capability. The OP said: > I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human. Since an actual human might be out to get me already the question is not motives but capability - should I fear evil AIs more than I fear evil humans? Or an evil human with a friendly leashed AI? The argument typically revolves around whether or not AI would self-improve and lead to runaway intelligence, but I think that that's a distraction - there are plenty of other ways AI could be dangerous that are easier to persuade people with without running into the that-sounds-like-scifi filter.
jamii 11y ago
Also, to a first approximation capability == danger. Just look at what even well-meaning humans have managed to do - imagine an AI that could eg design memes more persuasive than Marxism. We can't even get humans to agree on ethical values, let alone explain them to an AI, so even a perfectly altruistic AI could still be a disaster if it has the wrong value system, or if we don't perfectly understand the consequences of our own values systems when fully enacted.