4 ms·

Warning: the (real world) point of the fanfic is to recruit people into a cult whose goal is to prevent evil artificial intelligence from taking over the world.

Warning: the (real world) point of the fanfic is to recruit people into a cult whose goal is to prevent evil artificial intelligence from taking over the world. Seriously.

kbenson 12y ago
You make that sound like a bad thing?
- shinigami 12y ago
  Paved with good intentions and so on...
  - TeMPOraL 12y ago
    Well I hope that at least Google and Facebook guys pay attention, lest they unleash a recursively self-improving hell on us all ;).
TeMPOraL 12y ago
> to prevent evil artificial intelligence from taking over the world Many smart people find good reasons to care about that. But even if it's a personal goal for the author, it doesn't mean it's what he wants other people to do. http://lesswrong.com/lw/66/rationality_common_interest_of_many_causes/ http://lesswrong.com/lw/66/rationality_common_interest_of_ma... > into a cult Oh well, you realize you're writing this on another cult's news page? ;). http://lesswrong.com/lw/4d/youre_calling_who_a_cult_leader/ http://lesswrong.com/lw/4d/youre_calling_who_a_cult_leader/
- shinigami 12y ago
  > I've never heard of Paul Graham saying or doing a single thing that smacks of cultishness I love how he implicitly acknowledges that he does a lot of things that smacks of cultishness :) But yes, believe it or not, I was the first person who posted "Paul Graham Ate Breakfast" as a comment in reddit, and someboy created a fake news webpage and submitted it.
  - TeMPOraL 12y ago
    > I love how he implicitly acknowledges that he does a lot of things that smacks of cultishness :) Well he does and doesn't hide from it, and the "Eliezer Cult" is a running joke on LessWrong. Someone close to him even made a cover for his writings [0] ;). Most of his writings are basic maths + some good thinking, but there are points where he holds opinions many find eccentric. He also discusses cults at length, you might find [1] interesting ;). [0] - http://hg.ciphergoth.org/scrape-sequences/raw-file/8504183da617/the_book_2.jpg http://hg.ciphergoth.org/scrape-sequences/raw-file/8504183da... [1] - http://lesswrong.com/lw/md/cultish_countercultishness/ http://lesswrong.com/lw/md/cultish_countercultishness/
nemo 12y ago
My favorite event in their utterly credulous certainty in the inevitability of a friendly AI was the response to 'Roko's Basilisk.' http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk No rational person could describe that groupthink madness as a cult. :)
- TeMPOraL 12y ago
  Funny there's always this one little thing one can bring up to point that a non-mainstream group is definitely a cult and thus are a bunch of loons :). I used to be involved with a religious group many outsiders consider to be a cult. There was always this event that everyone outside kept bringing up - that one little prediction 40 years ago that some people took too seriously and ended up making stupid life decisions. (I'm not close to that group anymore, but I still value those people as the most honest and consistent in their thinking folks I ever met) Yes, people who extrapolate things from their beliefs to guide their life decisions can and do make mistakes. They will reach stupid conclusions and believe them until they figure out that those conclusions are stupid and rethink things again. I still respect them more than the usual crowd (both in religious and non-religious groups) who claim to believe (or believe that they believe) one thing and actually believe something else. I value consistency in thinking. EDIT: Disclaimer, before I get labelled as a cult priest for replying in the cult thread for the fourth time: I have my pet peeve with cult accusations, because I've been close with many groups accused of cultishness - both religious and secular, and I know how this is an impossible accusation to defend from. If you try to tell "hey, it's not a cult, because X" then it's definitely a cult, because that's obviously what a cult member would say. Also, you want to see real cults? Try political parties ;).
  - swombat 12y ago
    There's some pretty salient characteristics of an actual cult that make it different from just a group of very devoted followers. Namely: - Cults usually employ a number of heavy-handed psychological manipulation techniques, often together (e.g. 12-hour conference sessions; speakers that drone on at at about 50 beats per minute; loud, lengthy chanting that exhausts your brain's oxygen and makes you less critical; preying on people who are desperate for some reason) - It's not enough for a cult to have you as a member - they also have to remove the influence of other movements from your life, lest you might realise who stupid it all is and break off. So part of a cult's mode of control will involve breaking off your relationships with other groups important in your life: family, religion, work... and replacing them with social ties within the cult. - Cults generally require escalating levels of contribution. At first maybe you just need to donate a bit of money, but by the end your entire life is at the service of the cult. If the accused cult doesn't fit any of these criteria, it's probably not a cult, just a bunch of very devoted zealots. If it fits even one of these criteria, start to worry. If it fits more than one, it's almost certainly a "bad" cult. Source: my dad's a psychologist and studied cults, but this is just from memory.
loup-vaillant 12y ago
You make it sound like AI is not a serious risk… I'm not going to justify myself (okay, a little bit[1]), but I must say I'm at a point where such assumptions make me raise an eyebrow. [1] http://intelligenceexplosion.com/ http://intelligenceexplosion.com/
- shinigami 12y ago
  It seems it may be a risk, but we are nowhere near building it.
  - loup-vaillant 12y ago
    That is hard to assess. Apparently, AI requires more insights than hard work. Hard work is predictable, and gives us things like Moore's law. Insight on the other hand, can emerge at any time. When we solve those insights, we will be more able to accurately estimate when AI is due. It may be no time at all however, if hardware proves to be capable enough already.
lurkinggrue 12y ago
Yeah, I'm all for the evil AI taking over.

Warning: the (real world) point of the fanfic is to recruit people into a cult whose goal is to prevent evil artificial intelligence from taking over the world. Seriously.

kbenson 12y ago

You make that sound like a bad thing?

shinigami 12y ago
Paved with good intentions and so on...
- TeMPOraL 12y ago
  Well I hope that at least Google and Facebook guys pay attention, lest they unleash a recursively self-improving hell on us all ;).

TeMPOraL 12y ago

> to prevent evil artificial intelligence from taking over the world Many smart people find good reasons to care about that. But even if it's a personal goal for the author, it doesn't mean it's what he wants other people to do. http://lesswrong.com/lw/66/rationality_common_interest_of_many_causes/ http://lesswrong.com/lw/66/rationality_common_interest_of_ma... > into a cult Oh well, you realize you're writing this on another cult's news page? ;). http://lesswrong.com/lw/4d/youre_calling_who_a_cult_leader/ http://lesswrong.com/lw/4d/youre_calling_who_a_cult_leader/

shinigami 12y ago
> I've never heard of Paul Graham saying or doing a single thing that smacks of cultishness I love how he implicitly acknowledges that he does a lot of things that smacks of cultishness :) But yes, believe it or not, I was the first person who posted "Paul Graham Ate Breakfast" as a comment in reddit, and someboy created a fake news webpage and submitted it.
- TeMPOraL 12y ago
  > I love how he implicitly acknowledges that he does a lot of things that smacks of cultishness :) Well he does and doesn't hide from it, and the "Eliezer Cult" is a running joke on LessWrong. Someone close to him even made a cover for his writings [0] ;). Most of his writings are basic maths + some good thinking, but there are points where he holds opinions many find eccentric. He also discusses cults at length, you might find [1] interesting ;). [0] - http://hg.ciphergoth.org/scrape-sequences/raw-file/8504183da617/the_book_2.jpg http://hg.ciphergoth.org/scrape-sequences/raw-file/8504183da... [1] - http://lesswrong.com/lw/md/cultish_countercultishness/ http://lesswrong.com/lw/md/cultish_countercultishness/

nemo 12y ago

My favorite event in their utterly credulous certainty in the inevitability of a friendly AI was the response to 'Roko's Basilisk.' http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk No rational person could describe that groupthink madness as a cult. :)

TeMPOraL 12y ago
Funny there's always this one little thing one can bring up to point that a non-mainstream group is definitely a cult and thus are a bunch of loons :). I used to be involved with a religious group many outsiders consider to be a cult. There was always this event that everyone outside kept bringing up - that one little prediction 40 years ago that some people took too seriously and ended up making stupid life decisions. (I'm not close to that group anymore, but I still value those people as the most honest and consistent in their thinking folks I ever met) Yes, people who extrapolate things from their beliefs to guide their life decisions can and do make mistakes. They will reach stupid conclusions and believe them until they figure out that those conclusions are stupid and rethink things again. I still respect them more than the usual crowd (both in religious and non-religious groups) who claim to believe (or believe that they believe) one thing and actually believe something else. I value consistency in thinking. EDIT: Disclaimer, before I get labelled as a cult priest for replying in the cult thread for the fourth time: I have my pet peeve with cult accusations, because I've been close with many groups accused of cultishness - both religious and secular, and I know how this is an impossible accusation to defend from. If you try to tell "hey, it's not a cult, because X" then it's definitely a cult, because that's obviously what a cult member would say. Also, you want to see real cults? Try political parties ;).
- swombat 12y ago
  There's some pretty salient characteristics of an actual cult that make it different from just a group of very devoted followers. Namely: - Cults usually employ a number of heavy-handed psychological manipulation techniques, often together (e.g. 12-hour conference sessions; speakers that drone on at at about 50 beats per minute; loud, lengthy chanting that exhausts your brain's oxygen and makes you less critical; preying on people who are desperate for some reason) - It's not enough for a cult to have you as a member - they also have to remove the influence of other movements from your life, lest you might realise who stupid it all is and break off. So part of a cult's mode of control will involve breaking off your relationships with other groups important in your life: family, religion, work... and replacing them with social ties within the cult. - Cults generally require escalating levels of contribution. At first maybe you just need to donate a bit of money, but by the end your entire life is at the service of the cult. If the accused cult doesn't fit any of these criteria, it's probably not a cult, just a bunch of very devoted zealots. If it fits even one of these criteria, start to worry. If it fits more than one, it's almost certainly a "bad" cult. Source: my dad's a psychologist and studied cults, but this is just from memory.

loup-vaillant 12y ago

You make it sound like AI is not a serious risk… I'm not going to justify myself (okay, a little bit[1]), but I must say I'm at a point where such assumptions make me raise an eyebrow. [1] http://intelligenceexplosion.com/ http://intelligenceexplosion.com/

shinigami 12y ago
It seems it may be a risk, but we are nowhere near building it.
- loup-vaillant 12y ago
  That is hard to assess. Apparently, AI requires more insights than hard work. Hard work is predictable, and gives us things like Moore's law. Insight on the other hand, can emerge at any time. When we solve those insights, we will be more able to accurately estimate when AI is due. It may be no time at all however, if hardware proves to be capable enough already.

lurkinggrue 12y ago

Yeah, I'm all for the evil AI taking over.