3 ms·

No doubt that his "inside arguments" have been rebutted extensively by the strong AI optimists and their singularity priests. After all, dreaming up scenarios i

by tboyd47 10y ago

No doubt that his "inside arguments" have been rebutted extensively by the strong AI optimists and their singularity priests. After all, dreaming up scenarios in which robotic superintelligence dominates humanity is their version of saving the world. That's why I found the "outside arguments" here equally important and compelling. > The outside view doesn't care about content, it sees the form and the context, and it doesn't look good. If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. The best rebuttals to all this are the least engaging. "Dude, are you telling me you want to build Skynet?"

lmm 10y ago
> If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. Occasionally the crazies are right. Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk, except turns out they were actually doing this the whole time. The fact that the world hasn't ended tells us virtually nothing about how likely the end of the world is, for the simple reason that if the world had ended we wouldn't be here to talk about it. So we can't take it as evidence, at least, not conclusive evidence. Note also that the same argument works just as well against global warming as it does against AI risk. Turn it around. Suppose there was a genuinely real risk of destroying the world. How would you tell? How would you distinguish between the groups that had spotted the real danger and the run-of-the-mill end-of-the-world cults?
- tboyd47 10y ago
  The point that I was trying to bring attention to is that one's perception of the AI risk movement changes substantially once you turn your focus from content to form and context. He brings many examples of this (the cult in the robes, Art Bell, etc.) I wouldn't call suspecting the government of surveilling people "crazy." Claiming you know the timeframe for the apocalypse with precision is different. I am from North Carolina -- try a Google Image search for "May 21, 2011." Ray Kurzweil believes he will never die. Balancing content with form and context, what we have here is clearly an atheistic, scientist version of "May 21, 2011."
- eli_gottlieb 10y ago
  >Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk No, I remember the time that we'd known there were various signal intelligence programs operated by the US government for decades, and it was just a matter of guessing what the next and biggest one would be.
novaleaf 10y ago
The posters rebuttals against the threat are totally under-thought cop-outs. For example, his first argument about "how would Hawking get his cat in a cage?" just put food in it. It's not hard to imagine an AI could come up with a similar motivation to get humans to do what it wants. That's not to say that his general premise is wrong, but it's hard for me to take it seriously when his rebuttals are this weak.
- CydeWeys 10y ago
  The emu "war" example is also similarly dumb. We know that humanity is very successful at wiping out animals even when we're not actively trying, just as a consequence of habitat encroachment or over-hunting. If you want to kill a bunch of emus, that can easily be done using appropriate methods. Having a cohesive military formation go after them in the huge Australian outback and giving up after a week is not the way to do so.

by tboyd47 10y ago

lmm 10y ago

> If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. Occasionally the crazies are right. Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk, except turns out they were actually doing this the whole time. The fact that the world hasn't ended tells us virtually nothing about how likely the end of the world is, for the simple reason that if the world had ended we wouldn't be here to talk about it. So we can't take it as evidence, at least, not conclusive evidence. Note also that the same argument works just as well against global warming as it does against AI risk. Turn it around. Suppose there was a genuinely real risk of destroying the world. How would you tell? How would you distinguish between the groups that had spotted the real danger and the run-of-the-mill end-of-the-world cults?

tboyd47 10y ago
The point that I was trying to bring attention to is that one's perception of the AI risk movement changes substantially once you turn your focus from content to form and context. He brings many examples of this (the cult in the robes, Art Bell, etc.) I wouldn't call suspecting the government of surveilling people "crazy." Claiming you know the timeframe for the apocalypse with precision is different. I am from North Carolina -- try a Google Image search for "May 21, 2011." Ray Kurzweil believes he will never die. Balancing content with form and context, what we have here is clearly an atheistic, scientist version of "May 21, 2011."
eli_gottlieb 10y ago
>Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk No, I remember the time that we'd known there were various signal intelligence programs operated by the US government for decades, and it was just a matter of guessing what the next and biggest one would be.

novaleaf 10y ago

The posters rebuttals against the threat are totally under-thought cop-outs. For example, his first argument about "how would Hawking get his cat in a cage?" just put food in it. It's not hard to imagine an AI could come up with a similar motivation to get humans to do what it wants. That's not to say that his general premise is wrong, but it's hard for me to take it seriously when his rebuttals are this weak.

CydeWeys 10y ago
The emu "war" example is also similarly dumb. We know that humanity is very successful at wiping out animals even when we're not actively trying, just as a consequence of habitat encroachment or over-hunting. If you want to kill a bunch of emus, that can easily be done using appropriate methods. Having a cohesive military formation go after them in the huge Australian outback and giving up after a week is not the way to do so.