5 ms·

The cult of 'AI risk' is a religious belief that brooks no dissent, and unfortunately has its claws in a lot of otherwise smart people.

nbouscal 10y ago
The cult of 'AI will be totally fine' is a religious belief that brooks no dissent, and unfortunately has its claws in a lot of otherwise smart people.
- backpropaganda 10y ago
  That's a strawman. People who scoff at the "AI risk" cult don't believe that "AI will be totally fine". They just believe that it doesn't deserve the undue amount of attention it gets, which distracts from other more important and related risks, such as what Cambridge Analytica did for Trump, fake news due to internet, Facebook's bubbles, etc. The belief is to solve problems we have right now, or will have in 10-20 years, not the problems we'd have in 100 years.
  - nbouscal 10y ago
    I've been personally told by multiple different people who scoff at AI risk that it will be totally fine, so it's not a straw man. The median expert estimate for when we'll be 10% likely to have human-level AI is ~10 years. AI risk research didn't receive a penny of funding until the last few years, and is still funded at way lower levels than a lot of things that have dramatically less impact. In nearly every debate on the topic I've seen (with a few exceptions), the people concerned about AI risk have carefully considered the topic, are aware of the areas where there's still a lot of uncertainty, and make clear and well-hedged arguments that acknowledge that uncertainty; meanwhile the people who scoff at it haven't read any of the arguments (not even in popular book form in Superintelligence), haven't thought about most of the considerations, and have a general air of "assuming things will probably be fine". That's not a straw man, that's just direct observation of the state of the debate. People are doing serious academic work on the topic and have thought about it very deeply; the standard HN middlebrow dismissal is both common and inappropriate.
    - icebraining 10y ago
      That number didn't pass my sniff test, so I went looking. It seems to have come from here[1], which aggregates a series of surveys. I first opened the "FHI Winter Intelligence" report: it's an informal survey made to 35 participants of a conference, of which only 8 work on AI at all (let alone be an expert in AGI). I then looked at the "Kruel interviews", which the site reports as giving a prediction of "2025" for 10% chance, yet reading the interviews it's quite clear that many gave no prediction at all. Also, averaging answers by people ranging from Pat Heyes to PhD students seems suspect. Is your number based on these reports? [1] http://aiimpacts.org/ai-timeline-surveys/ http://aiimpacts.org/ai-timeline-surveys/
      - nbouscal 10y ago
        Sorry, gave a citation in another comment on the thread but not in this one. I was referencing http://sophia.de/pdf/2014_PT-AI_polls.pdf http://sophia.de/pdf/2014_PT-AI_polls.pdf
        icebraining 10y ago
        Are they actually experts, though? From that paper: “Concerning the above questions, how would you describe your own expertise?” (0 = none, 9 = expert) − Mean 5.85 “Concerning technical work in artificial intelligence, how would you describe your own expertise?” (0 = none, 9 = expert) − Mean 6.26 Also, the whole methodology of aggregating the opinions of random conference attendees seems suspect to me. Attending a conference doesn't make you an expert.
        nbouscal 10y ago
        You can restrict your attention to the TOP100 group if you prefer.
        icebraining 10y ago
        Yeah, but then you just have 29 responses in total.
Noos 10y ago
You can't get rid of human religious longing easily. Instead of the rapture and armageddon, we have transhumanism and ai risk.
- jaibot 10y ago
  Fun fact: You can apply this template to anyone worried about any potential disaster. "Behold the cult of global warming, who believe that the unclean practices of man will anger the sky gods and bring down furious vengeance. Sad to see so many otherwise-smart climatologists drawn in by this drivel." "'Antibiotic resistance'? Oh my god, you must be one of those 'biosecurity cultists'. Do you really believe that magical microscopic beings are going to grow strong because we're feeding farm animals the wrong food, and that those same invisible creatures will destroy human civilization? Oh man, do you have chants? What a nutjob." "Seriously, you're really afraid of 'nuclear weapons'? You really believe that people will somehow cast some magic spell that will turn rocks into fire and destroy entire cities, that this same magic will cause 'mutations' and a 'nuclear winter'? Can't you see that you're part of a doomsday cult?"
  - icebraining 10y ago
    Thing is, we have good evidence that all those things actually happen (except maybe the nuclear winter), and that they are/were increasing. Meanwhile, all we have from the AGI doomsday people is sci-fi stories, and actual programs that still can barely distinguish a cat from a mole even after looking at thousands of pictures.
    - jaibot 10y ago
      It would be really useful if there were some historical example of a sudden increase in intelligence leading to a new class of agents taking over the Earth and determining it's future. Or maybe if there were any evidence that AI capabilities are increasing, like becoming dominant at Chess or Go, driverless cars, or a slew of recent papers on transfer learning. Maybe if one of the co-authors of the leading AI textbook, Stuart Russell, voiced concerns, we could count that as evidence. But you're right, it's better to wait until we know for a fact that someone's built an agent smart enough to end civilization before we commit any resources to the problem.
Jach 10y ago
Ways in which it resembles a religious belief: proponents argue passionately, seem impossible to be dissuaded even in the face of ridicule, and talk about some sort of transcendence beyond the normal human condition; some get together and chant litanies. Ways it is different: obsession over cognitive heuristics and biases, AI is not supernatural and must be human caused, behavior not totally certain (it is "risk" after all), no perks that true believers receive should their hopes and wishes come true that non-believers or apostates don't also receive nor revenge fantasies of such, no requirement for tithing, no Big Alpha divine or otherwise to represent or interpret or decree official approved beliefs (though I'll admit Big Yud can seem close), no anthropomorphizing the AI, and proponents concede evidence not faith must determine beliefs and actions so are willing to look for such.
jaibot 10y ago
I'm old enough to remember when "cult" didn't just mean "people who have high confidence in ideas that seem weird to me".

nbouscal 10y ago

The cult of 'AI will be totally fine' is a religious belief that brooks no dissent, and unfortunately has its claws in a lot of otherwise smart people.

backpropaganda 10y ago
That's a strawman. People who scoff at the "AI risk" cult don't believe that "AI will be totally fine". They just believe that it doesn't deserve the undue amount of attention it gets, which distracts from other more important and related risks, such as what Cambridge Analytica did for Trump, fake news due to internet, Facebook's bubbles, etc. The belief is to solve problems we have right now, or will have in 10-20 years, not the problems we'd have in 100 years.
- nbouscal 10y ago
  I've been personally told by multiple different people who scoff at AI risk that it will be totally fine, so it's not a straw man. The median expert estimate for when we'll be 10% likely to have human-level AI is ~10 years. AI risk research didn't receive a penny of funding until the last few years, and is still funded at way lower levels than a lot of things that have dramatically less impact. In nearly every debate on the topic I've seen (with a few exceptions), the people concerned about AI risk have carefully considered the topic, are aware of the areas where there's still a lot of uncertainty, and make clear and well-hedged arguments that acknowledge that uncertainty; meanwhile the people who scoff at it haven't read any of the arguments (not even in popular book form in Superintelligence), haven't thought about most of the considerations, and have a general air of "assuming things will probably be fine". That's not a straw man, that's just direct observation of the state of the debate. People are doing serious academic work on the topic and have thought about it very deeply; the standard HN middlebrow dismissal is both common and inappropriate.
  - icebraining 10y ago
    That number didn't pass my sniff test, so I went looking. It seems to have come from here[1], which aggregates a series of surveys. I first opened the "FHI Winter Intelligence" report: it's an informal survey made to 35 participants of a conference, of which only 8 work on AI at all (let alone be an expert in AGI). I then looked at the "Kruel interviews", which the site reports as giving a prediction of "2025" for 10% chance, yet reading the interviews it's quite clear that many gave no prediction at all. Also, averaging answers by people ranging from Pat Heyes to PhD students seems suspect. Is your number based on these reports? [1] http://aiimpacts.org/ai-timeline-surveys/ http://aiimpacts.org/ai-timeline-surveys/
    - nbouscal 10y ago
      Sorry, gave a citation in another comment on the thread but not in this one. I was referencing http://sophia.de/pdf/2014_PT-AI_polls.pdf http://sophia.de/pdf/2014_PT-AI_polls.pdf
      - icebraining 10y ago
        Are they actually experts, though? From that paper: “Concerning the above questions, how would you describe your own expertise?” (0 = none, 9 = expert) − Mean 5.85 “Concerning technical work in artificial intelligence, how would you describe your own expertise?” (0 = none, 9 = expert) − Mean 6.26 Also, the whole methodology of aggregating the opinions of random conference attendees seems suspect to me. Attending a conference doesn't make you an expert.
        nbouscal 10y ago
        You can restrict your attention to the TOP100 group if you prefer.
        icebraining 10y ago
        Yeah, but then you just have 29 responses in total.

Noos 10y ago

You can't get rid of human religious longing easily. Instead of the rapture and armageddon, we have transhumanism and ai risk.

jaibot 10y ago
Fun fact: You can apply this template to anyone worried about any potential disaster. "Behold the cult of global warming, who believe that the unclean practices of man will anger the sky gods and bring down furious vengeance. Sad to see so many otherwise-smart climatologists drawn in by this drivel." "'Antibiotic resistance'? Oh my god, you must be one of those 'biosecurity cultists'. Do you really believe that magical microscopic beings are going to grow strong because we're feeding farm animals the wrong food, and that those same invisible creatures will destroy human civilization? Oh man, do you have chants? What a nutjob." "Seriously, you're really afraid of 'nuclear weapons'? You really believe that people will somehow cast some magic spell that will turn rocks into fire and destroy entire cities, that this same magic will cause 'mutations' and a 'nuclear winter'? Can't you see that you're part of a doomsday cult?"
- icebraining 10y ago
  Thing is, we have good evidence that all those things actually happen (except maybe the nuclear winter), and that they are/were increasing. Meanwhile, all we have from the AGI doomsday people is sci-fi stories, and actual programs that still can barely distinguish a cat from a mole even after looking at thousands of pictures.
  - jaibot 10y ago
    It would be really useful if there were some historical example of a sudden increase in intelligence leading to a new class of agents taking over the Earth and determining it's future. Or maybe if there were any evidence that AI capabilities are increasing, like becoming dominant at Chess or Go, driverless cars, or a slew of recent papers on transfer learning. Maybe if one of the co-authors of the leading AI textbook, Stuart Russell, voiced concerns, we could count that as evidence. But you're right, it's better to wait until we know for a fact that someone's built an agent smart enough to end civilization before we commit any resources to the problem.

Jach 10y ago

Ways in which it resembles a religious belief: proponents argue passionately, seem impossible to be dissuaded even in the face of ridicule, and talk about some sort of transcendence beyond the normal human condition; some get together and chant litanies. Ways it is different: obsession over cognitive heuristics and biases, AI is not supernatural and must be human caused, behavior not totally certain (it is "risk" after all), no perks that true believers receive should their hopes and wishes come true that non-believers or apostates don't also receive nor revenge fantasies of such, no requirement for tithing, no Big Alpha divine or otherwise to represent or interpret or decree official approved beliefs (though I'll admit Big Yud can seem close), no anthropomorphizing the AI, and proponents concede evidence not faith must determine beliefs and actions so are willing to look for such.

jaibot 10y ago

I'm old enough to remember when "cult" didn't just mean "people who have high confidence in ideas that seem weird to me".