7 ms·

The Most Terrifying Thought Experiment of All Time

36 points by pnp 12y ago

danepowell 12y ago
If we're going to talk about Yudkowsky and boxes, the AI-Box experiment is much more interesting than this silly "basilisk" nonsense: http://yudkowsky.net/singularity/aibox/ http://yudkowsky.net/singularity/aibox/
bungle 12y ago
Just don't choose any of the boxes and be done with it, ;-). It will BSOD the God machine.
- Liesmith 12y ago
  Flip a coin.
  - thret 12y ago
    That was my thought too, until I realised the only correct response for the AI would be to also flip a coin. I'd prefer to take the million rather than a 50% chance to beat the AI.
    - danielweber 12y ago
      I'm a one-boxer because $1,000,000 is a lot of money to gamble that I would be the (first? only? one of the few?) person who beats the system. I also wouldn't take on Kasparov in a chess match or John McEnroe in a tennis match at even odds; I definitely wouldn't put $1,000,000 on the line to potentially increase my winnings to $1,001,000.
- vaidhy 12y ago
  They forgot to add the key element.. Not choosing one will end up in eternal torment too..
- danielweber 12y ago
  I choose box One'); -- DROP TABLE History;.
spingsprong 12y ago
Nobody is going to make a malevolent universe controlling AI on purpose. If a massively powerful and knowledgeable AI was created by accident, it would have nothing to gain by being malevolent. And both those points don't matter anyway, since the technological singularity is impossible.
- dm2 12y ago
  Hopefully your first statement is true, but I've seen individuals do really dumb stuff. A true AI would have the ability to learn and to evolve. Watch the movie Transcendence, I see very few flaws in that movie's logic. If humans posed a threat to the AI it could certainly be malevolent from our perspective, to it it would just be doing what's necessary to survive. Computers were "impossible" 200 years ago, the internet was something that few people could even imagine. I don't really consider anything impossible, what would make it truly impossible given enough time, resources, and motivation?
  - JetSpiegel 12y ago
    The Internet could certainly be imagined 200 years ago. It's just a series of interconnected telegraph lines.
    - dm2 12y ago
      Even considering telegraphs only input and output was clicks? No wireless, no converting the clicks to anything but words using Morse Code. Was it really reasonable back then to imagine wires running to every house? Electronic signals all around us providing data constantly? Today I can stand almost anywhere (ignoring cell coverage and different technologies required to use satellite phones) and talk to any other human on the planet with video chat. I can instantly access any and all information online that's being sent to me from millions of servers. It still blows my mind that it's actually been implemented in such a small amount of time, and I grew up with it. I can barely believe we have electricity, water, sewage, internet, and phone lines running to 99+% of the houses and businesses in the country, in most countries even.
zeeed 12y ago
here's the comment thread from when the topic last surfaced 6 days ago: https://news.ycombinator.com/item?id=8051533 https://news.ycombinator.com/item?id=8051533
dm2 12y ago
TL,DR: Here is the thought experiment, 4th paragraph after 3 long paragraphs of stuff I didn't want to read. "What if, in the future, a somewhat malevolent AI were to come about and punish those who did not do its bidding? What if there were a way (and I will explain how) for this AI to punish people today who are not helping it come into existence later? In that case, weren’t the readers of LessWrong right then being given the choice of either helping that evil AI come into existence or being condemned to suffer?" I don't think it's that interesting of a thought experiment, there are dozens of much more insightful and thought-provoking ones out there. Search for "best thought experiments" on Google if you want some cool things to think about. It might be taboo to mention but weed will make thought experiments really fun, in my opinion.
- resdirector 12y ago
  OT but IMO it's such a huge cost to the economy (and humanity) that marijuana is still taboo. The thought experiments that you describe, for me at least, are not only fun, but more often than not lead to practical, wealth-generating breakthroughs. I assume I'm not the only one. I hope one day historians and economists will study the lost opportunity costs of the prohibition / taboo nature of marijuana.
  - dm2 12y ago
    Richard Feynman, Carl Sagan, Bill Gates, Bill Clinton, Richard Branson and thousands of other top minds would agree with what you said. I don't think anyone would actually disagree and be able to provide a valid reason. Politics, money, religion, and several other irrelevant factors keep it highly illegal. It even has come to the point where states are ignoring federal law, so progress is happening thankfully. Changing the leadership of the DEA would be helpful. Remove guns from the DEA and require them to use the FBI or local SWAT if a crime is bad enough to warrant a raid. "They might flush the evidence" is not a good enough reason to raid good peoples houses and suppress something that does far more good than harm. How many people have died from marijuana and how many people have died from a marijuana related DEA raid? Which is really more harmful to the country? Money seized should go towards public education, not directly to the law-enforcement agencies. Is it off topic? We are talking about thought-experiments here, and your experience is normal for anyone with even a slightly scientific mind.
teekert 12y ago
"it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes." This makes him Yudkowsky the perfect lab animal! Think of the things we can learn, the lives we can save by performing experiments (that most think are unethical) on him! As for the box choice? Just start chopping through the alien's head, the unknown that lies beyond will create new input for the simulation. Who knows what nice benevolent digital organism named Jane might sprout from this in the future?! (Edit: This is an Enders game/Speaker for the Dead reference, Ender also find himself presented with an impossible choice in a simulation at some point.) Anyway, worrying about this will not make your life better, nor will it make your kids' life better. What is life all about according to this guy? I hope for the people that love him that he will get his priorities straight.
- onion2k 12y ago
  This makes him Yudkowsky the perfect lab animal! Think of the things we can learn, the lives we can save by performing experiments (that most think are unethical) on him! Only if you have a way to stop the other people getting dust in their eyes. Otherwise you're merely increasing the amount of suffering rather than mitigating it by transferring it all to one person.
  - teekert 12y ago
    Of course, we'd have to be sure first. But, perhaps we can start right away with splitting his organs among people who are on the waiting lists, surely, just with his kidneys we can already increase the life of two persons more than the kidneys can increase his life alone. Imagine if we harvest all of his organs! What a guy and what a theory this moral utilitarianism! It's great! Think of the possibilities, they're endless. I always thought that the only one who could put a value on my life was me. But this moral utilitarianism makes things so much simpler.
    - FeepingCreature 12y ago
      Honestly? If there was an insurance where I could sign up and they'd optionally harvest my organs if they'd find a really lopsided situation like "save those other three people and you're a really good bodytype match", I'd sign up for that if it gave me better expected lifetime than regular approaches. - Just because it's taboo, doesn't mean it's bad. Just because you consider it absurd doesn't mean that others would.
jpeterson 12y ago
Well, that did it. Lifetime ignore for LessWrong or anything LessWrong-related. I want my 10 minutes back.
DanielBMarkham 12y ago
What a cute story. A logical Chinese finger puzzle. Seems like this argument isn't new, is it? Those folks who both believe in a watchmaker's universe and freewill at the same time are also always going on about how critical it is to only make the "correct" choices. It's circular reasoning. If a being exists that can tell which choices you are going to make, and it is able to kill you or prevent you from making those choices ahead of time, and some of those choices involve its existence? Then you'll ever be able to make choices that cause it not to exist. Or, with a bit of finer detail, the aggregate of all the choices made will never be such that it does not exist. Since the predicate was that such a being exists, we have returned to our starting point. Yadda yadda. In short, "How can something not exist that must exist?"
aaronem 12y ago
It's a shame no one there seems to have noticed that they reinvented Yog-Sothoth from first principles.
paul_f 12y ago
Good grief that was a waste of time. This entire concept only works if time travel is possible and will be discovered in the future. It isn't, and it won't be.
- teekert 12y ago
  Time travel... or something like Asimov's Psychohistory (http://en.wikipedia.org/wiki/Psychohistory_%28fictional%29 http://en.wikipedia.org/wiki/Psychohistory_%28fictional%29) Granted that does not have anything to do with the box choice.
- brunnock 12y ago
  I think the fear is that the Basilisk will have access to our past records and will be able to determine if we've been naughty or nice.
- darkFunction 12y ago
  It's based on the argument that a simulation of yourself is not just an exact copy but actually you and that you will experience what the simulation experiences. I remember reading about an interesting thought experiment where we imagine two separate people able to experience each others' perspective through some technology, being able to switch at will. The author was able to gradually blur the lines between both conciousnesses to really make you confused about where one mind ended and another began. I'm not doing a very good job of explaining, but I think that's what they are building on here. I agree though, it's stupid.
- cousin_it 12y ago
  How so? The future AI just needs to have access to the financial records of the organization that created it, so it can find and punish the folks who didn't donate enough. No time travel involved, no brain simulations, nothing. Actually, Roko's original version of the basilisk was a pretty nice AI that wanted to help people. A hundred people die every minute, how many of them could be saved with better technology? If donating your entire net worth would speed up the AI's creation by one day, that could be very worthwhile, according to the combined wishes of all humanity (which are encoded in the AI's utility function). Threatening to punish you is a comparatively small price to pay. Especially if you decide to comply, and the punishment never actually happens! At this point you might be indignantly asking, why would the AI decide to punish you in the future? After all, that won't help the AI in the future, because its history will already be fixed. But that doesn't matter. The AI's algorithm tries to choose the most efficient decision overall, from a "position of ignorance", rather than the most efficient decision at a particular moment in time. Technically, the algorithm tries to find the best input-output mapping according to certain criteria, not find the best output for the particular input it happened to receive. There's nothing especially futuristic about such algorithms either, you can implement one as an ordinary Python program today (for toy problems of course).
lxmorj 12y ago
So stupid. Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable. When I was a kid I watched a movie about a monster that came up from the basement when the kids in the house accidentally said an ancient Indian incantation. "BlahBlahBluey" or something I got scared that perhaps, the last thing I said "Good Night, Batman" or whatever, was in fact coincidentally an ancient curse incantation and I had summoned a monster who was en route to kill me. "Blah!" I'd say, changing my words to "Good Night, Batman Blah!" Ah, much better. But then FUCK! What if "Good Night, Batman Blah!" is the cursed incantation!!! Repeat over and over until I was sufficiently convinced that no one had ever uttered the previous sequence of sounds I had just made, become exhausted from an entirely too powerful imagination, and fall asleep. I'm pretty sure this 'thought experiment' is about as intellectual as I was at bedtime when I was 7.
- teekert 12y ago
  "So stupid. Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable." I think the latter is a lot more likely and I predict we'll see it before the singularity.
  - probably_wrong 12y ago
    Does this machine count? http://betabeat.com/2014/06/crowdfunded-electronic-blowjob-machine-is-now-available-for-pre-order/ http://betabeat.com/2014/06/crowdfunded-electronic-blowjob-m... (warning: language mildly NSFW, plus of course a video of a blowjob machine. No naked ladies, though)
- sirodoht 12y ago
  > Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable. I agree. So what would they be so afraid of? My assumption is that the LessWrong community tries to come up with the best system of thinking, so by proving (by presenting a situation which shows that the best system is flawed, since we may be slaves right now) means that there really is no hope. As far as LessWrong in general, they share some controversial opinions but some of their concepts seem very interesting[1]. However, learning and especially thinking about these concepts does not appear to have any positive effect in everyday life. [1] http://lesswrong.com/lw/k5/cached_thoughts/ http://lesswrong.com/lw/k5/cached_thoughts/
iSnow 12y ago
How is this more than a souped-up version of Pascal's wager? You better believe in God now because if you don't and he exists, you'll rot in hell.
- Ono-Sendai 12y ago
  It sounds exactly like Pascal's wager to me, which means it suffers from all the same problems.
- jarman 12y ago
  Pascal's wager at least involves widely known and culturally influential concept. This is more like small kid drawing a bogeymen, being scared of it and throwing a tantrum
- Tenoke 12y ago
  Just in case anyone cares what the difference is - most possible Gods are about equally likely (no reason to think that one is much more likely to exist than another), while possible AIs which have reached SuperIntelligence status (to the point where they can simulate people for a start) are more likely to be certain ways (for example, self-preservation is not a consideration for an immortal God, nor is research something such a God must engage in, in order to gain knowledge, etc.).
  - davidgerard 12y ago
    As far as I can tell (and I've read LW since 2010), you're asserting that on no evidence.
    - Tenoke 12y ago
      I am giving the reasons why I think so, if you disagree, you can point out what is wrong in my line of thinking: AIs are more likely to care for self-preservation (in some way), because AIs that don't are less likely to exist (since they die earlier), while Gods could have other mechanisms for survival (such as being immortal right off the bat).
madaxe_again 12y ago
It seems that everyone deriding this as pointless may be unaware of the simulation argument. It hasn't been disproven, and it's more likely than not that at some point it will be proven, given the pace of technological advancement. When you subscribe to the idea that this could all be a sim, the whole thing results in mental contortions that are literally maddening. Baudrillard observed not dissimilar in Simulacrum.
- super_mario 12y ago
  The cost of simulating entire universe becomes increasingly expensive to the point you need a universe of energy to simulate a universe. Daniel Dennet in his "Consciousness Explained" makes a good argument why this is so. Also there was some research done on the constraints on the universe as a numerical simulation http://arxiv.org/abs/1210.1847 http://arxiv.org/abs/1210.1847 telling us that we can do experiments to detect if we are living in a simulation (but the simulators always have a freedom to "increase the resolution" of their simulation to thwart our efforts). The paper does not discuss if there are limits on how far this game can be played on either side.
  - madaxe_again 12y ago
    You don't need to simulate the entire universe in order to accurately simulate a causal chain - just the visible universe. Which is a lot smaller. Hell, you could just simulate an individual and what they observe. For all I know, I'm talking to the AI in which I, and solely I, reside right now. Woo, simulation-solipsism.
ryandvm 12y ago
I refuse to believe there are people stupid enough to fall for this tripe.
- davidgerard 12y ago
  You keep telling yourself that. I've had email from them. (It's not stupidity - as far as I can tell it's a combination of aspergism, OCD, a weak sense of self and reading way too much LessWrong and cut'n'pasting it into their heads. LW is a superstimulant for people of this description, which is how an organisation that really truly isn't trying to start a cult has inadvertently accumulated something around itself that is highly reminiscent of one.)
pessimizer 12y ago
Chick's Cthulhu: http://jackchick.wordpress.com/2009/07/08/chick-parody-who-will-be-eaten-first/ http://jackchick.wordpress.com/2009/07/08/chick-parody-who-w...
IgorPartola 12y ago
Really? Peter Thiel spends his time in this circle jerk? Let me give so life advice that will go unhedged, I am sure. If stuff like this gives you nightmares, get off your fucking computer. Get a different hobby so you can stop losing sleep over killer robots that you built just so they can torture you.
- Tenoke 12y ago
  There's about 1 confirmed person who got 'nightmares' from the basilisk (Roko), and possibly a very few others who have never said so publicly (as far as I know).
  - davidgerard 12y ago
    RationalWiki editors (including me ... and Dmytry, which is part of why he can be an asshole on LW) have had email from people traumatised by the basilisk. It turns out you can't think your way out of things you did think your way into, too. This is why the RationalWiki article http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk was created, because personal email doesn't scale.
    - IgorPartola 12y ago
      So people cannot just dismiss this thing as bullshit? What is the actual thinking of these individuals? I am having a really hard time relating to them.
      - davidgerard 12y ago
        From the small sample of victims I know of, they tend to be aspergic, have OCD, have not so strong a sense of self, and buy into all the LW versions of the transhumanist memes. They realise intellectually that it's a silly idea - but something about the story pushes all their buttons, perfectly wrongly. When I stsrted getting email from them (cos I edit RW, RW was pretty much the only place on the Internet that mentioned the basilisk at the time, and LW refused to talk about it so they couldn't voice their concerns there), and found others at RW had been getting email too, I decided I had to do something to help because email wouldn't scale but it would be inhuman not to try to help. Hence the RW article on the basilisk. No email since then, so it seems to do the job. See the first talk page archive for initial discussion as we try to work out how to talk about this completely fractally stupid thing in a manner that's helpful: http://rationalwiki.org/wiki/Talk:Roko%27s_basilisk/Archive1 http://rationalwiki.org/wiki/Talk:Roko%27s_basilisk/Archive1
adwn 12y ago
> LessWrong’s founder, Eliezer Yudkowsky, is a significant figure in techno-futurism [...] What? Is this true, or just gullible journalism? > Yudkowsky said that Roko had already given nightmares to several LessWrong users and had brought them to the point of breakdown. It seems like some users of LessWrong need to spend more time in the real world and less time on stupid thought experiments.
jcbrand 12y ago
There's something very religious about the whole singularity movement. Roko's Basilisk doesn't appear very different to me than Pascal's Wager, with the added twist that the God you have to believe in is actually a malevolent demiurge (as envisioned by the Gnostic christian sects more than a 1000 years ago). If you already have a cynical viewpoint on religion, then this thought experiment is very similar to the "deal" monotheistic religions have been offering through the ages. Either believe in our god (and therefore give us, as his sole representatives enormous, unfettered power), or suffer eternal hell! (or not ;) Throughout the ages, people were terrified of hell. This is just a modern twist on an old idea. I used to be scared of hell, until I realized that hell "existed" only as long I believed in it. If I stopped believing in it, it "disappeared". That's the power of belief. Yes, it might still exist whether I believe or not, but life is full of "what ifs", the contemplation of which might drive you crazy. Better to follow a positive and compassionate path then a fear based "what if" path.
thegeomaster 12y ago
I was actually more terrified by Smile.dog :)
DalekBaldwin 12y ago
We should all just commit to believing that Box A's label actually says "Devote your life to helping create Roko's Basilisk... who will arrive in the form of the Stay Puft Marshmallow Man".

36 points by pnp 12y ago

If we're going to talk about Yudkowsky and boxes, the AI-Box experiment is much more interesting than this silly "basilisk" nonsense: http://yudkowsky.net/singularity/aibox/ http://yudkowsky.net/singularity/aibox/

bungle 12y ago

Just don't choose any of the boxes and be done with it, ;-). It will BSOD the God machine.

Liesmith 12y ago
Flip a coin.
- thret 12y ago
  That was my thought too, until I realised the only correct response for the AI would be to also flip a coin. I'd prefer to take the million rather than a 50% chance to beat the AI.
  - danielweber 12y ago
    I'm a one-boxer because $1,000,000 is a lot of money to gamble that I would be the (first? only? one of the few?) person who beats the system. I also wouldn't take on Kasparov in a chess match or John McEnroe in a tennis match at even odds; I definitely wouldn't put $1,000,000 on the line to potentially increase my winnings to $1,001,000.
vaidhy 12y ago
They forgot to add the key element.. Not choosing one will end up in eternal torment too..
danielweber 12y ago
I choose box One'); -- DROP TABLE History;.

spingsprong 12y ago

Nobody is going to make a malevolent universe controlling AI on purpose. If a massively powerful and knowledgeable AI was created by accident, it would have nothing to gain by being malevolent. And both those points don't matter anyway, since the technological singularity is impossible.

dm2 12y ago
Hopefully your first statement is true, but I've seen individuals do really dumb stuff. A true AI would have the ability to learn and to evolve. Watch the movie Transcendence, I see very few flaws in that movie's logic. If humans posed a threat to the AI it could certainly be malevolent from our perspective, to it it would just be doing what's necessary to survive. Computers were "impossible" 200 years ago, the internet was something that few people could even imagine. I don't really consider anything impossible, what would make it truly impossible given enough time, resources, and motivation?
- JetSpiegel 12y ago
  The Internet could certainly be imagined 200 years ago. It's just a series of interconnected telegraph lines.
  - dm2 12y ago
    Even considering telegraphs only input and output was clicks? No wireless, no converting the clicks to anything but words using Morse Code. Was it really reasonable back then to imagine wires running to every house? Electronic signals all around us providing data constantly? Today I can stand almost anywhere (ignoring cell coverage and different technologies required to use satellite phones) and talk to any other human on the planet with video chat. I can instantly access any and all information online that's being sent to me from millions of servers. It still blows my mind that it's actually been implemented in such a small amount of time, and I grew up with it. I can barely believe we have electricity, water, sewage, internet, and phone lines running to 99+% of the houses and businesses in the country, in most countries even.

zeeed 12y ago

here's the comment thread from when the topic last surfaced 6 days ago: https://news.ycombinator.com/item?id=8051533 https://news.ycombinator.com/item?id=8051533

dm2 12y ago

TL,DR: Here is the thought experiment, 4th paragraph after 3 long paragraphs of stuff I didn't want to read. "What if, in the future, a somewhat malevolent AI were to come about and punish those who did not do its bidding? What if there were a way (and I will explain how) for this AI to punish people today who are not helping it come into existence later? In that case, weren’t the readers of LessWrong right then being given the choice of either helping that evil AI come into existence or being condemned to suffer?" I don't think it's that interesting of a thought experiment, there are dozens of much more insightful and thought-provoking ones out there. Search for "best thought experiments" on Google if you want some cool things to think about. It might be taboo to mention but weed will make thought experiments really fun, in my opinion.

resdirector 12y ago
OT but IMO it's such a huge cost to the economy (and humanity) that marijuana is still taboo. The thought experiments that you describe, for me at least, are not only fun, but more often than not lead to practical, wealth-generating breakthroughs. I assume I'm not the only one. I hope one day historians and economists will study the lost opportunity costs of the prohibition / taboo nature of marijuana.
- dm2 12y ago
  Richard Feynman, Carl Sagan, Bill Gates, Bill Clinton, Richard Branson and thousands of other top minds would agree with what you said. I don't think anyone would actually disagree and be able to provide a valid reason. Politics, money, religion, and several other irrelevant factors keep it highly illegal. It even has come to the point where states are ignoring federal law, so progress is happening thankfully. Changing the leadership of the DEA would be helpful. Remove guns from the DEA and require them to use the FBI or local SWAT if a crime is bad enough to warrant a raid. "They might flush the evidence" is not a good enough reason to raid good peoples houses and suppress something that does far more good than harm. How many people have died from marijuana and how many people have died from a marijuana related DEA raid? Which is really more harmful to the country? Money seized should go towards public education, not directly to the law-enforcement agencies. Is it off topic? We are talking about thought-experiments here, and your experience is normal for anyone with even a slightly scientific mind.

teekert 12y ago

"it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes." This makes him Yudkowsky the perfect lab animal! Think of the things we can learn, the lives we can save by performing experiments (that most think are unethical) on him! As for the box choice? Just start chopping through the alien's head, the unknown that lies beyond will create new input for the simulation. Who knows what nice benevolent digital organism named Jane might sprout from this in the future?! (Edit: This is an Enders game/Speaker for the Dead reference, Ender also find himself presented with an impossible choice in a simulation at some point.) Anyway, worrying about this will not make your life better, nor will it make your kids' life better. What is life all about according to this guy? I hope for the people that love him that he will get his priorities straight.

onion2k 12y ago
This makes him Yudkowsky the perfect lab animal! Think of the things we can learn, the lives we can save by performing experiments (that most think are unethical) on him! Only if you have a way to stop the other people getting dust in their eyes. Otherwise you're merely increasing the amount of suffering rather than mitigating it by transferring it all to one person.
- teekert 12y ago
  Of course, we'd have to be sure first. But, perhaps we can start right away with splitting his organs among people who are on the waiting lists, surely, just with his kidneys we can already increase the life of two persons more than the kidneys can increase his life alone. Imagine if we harvest all of his organs! What a guy and what a theory this moral utilitarianism! It's great! Think of the possibilities, they're endless. I always thought that the only one who could put a value on my life was me. But this moral utilitarianism makes things so much simpler.
  - FeepingCreature 12y ago
    Honestly? If there was an insurance where I could sign up and they'd optionally harvest my organs if they'd find a really lopsided situation like "save those other three people and you're a really good bodytype match", I'd sign up for that if it gave me better expected lifetime than regular approaches. - Just because it's taboo, doesn't mean it's bad. Just because you consider it absurd doesn't mean that others would.

jpeterson 12y ago

Well, that did it. Lifetime ignore for LessWrong or anything LessWrong-related. I want my 10 minutes back.

DanielBMarkham 12y ago

What a cute story. A logical Chinese finger puzzle. Seems like this argument isn't new, is it? Those folks who both believe in a watchmaker's universe and freewill at the same time are also always going on about how critical it is to only make the "correct" choices. It's circular reasoning. If a being exists that can tell which choices you are going to make, and it is able to kill you or prevent you from making those choices ahead of time, and some of those choices involve its existence? Then you'll ever be able to make choices that cause it not to exist. Or, with a bit of finer detail, the aggregate of all the choices made will never be such that it does not exist. Since the predicate was that such a being exists, we have returned to our starting point. Yadda yadda. In short, "How can something not exist that must exist?"

aaronem 12y ago

It's a shame no one there seems to have noticed that they reinvented Yog-Sothoth from first principles.

paul_f 12y ago

Good grief that was a waste of time. This entire concept only works if time travel is possible and will be discovered in the future. It isn't, and it won't be.

teekert 12y ago
Time travel... or something like Asimov's Psychohistory (http://en.wikipedia.org/wiki/Psychohistory_%28fictional%29 http://en.wikipedia.org/wiki/Psychohistory_%28fictional%29) Granted that does not have anything to do with the box choice.
brunnock 12y ago
I think the fear is that the Basilisk will have access to our past records and will be able to determine if we've been naughty or nice.
darkFunction 12y ago
It's based on the argument that a simulation of yourself is not just an exact copy but actually you and that you will experience what the simulation experiences. I remember reading about an interesting thought experiment where we imagine two separate people able to experience each others' perspective through some technology, being able to switch at will. The author was able to gradually blur the lines between both conciousnesses to really make you confused about where one mind ended and another began. I'm not doing a very good job of explaining, but I think that's what they are building on here. I agree though, it's stupid.
cousin_it 12y ago
How so? The future AI just needs to have access to the financial records of the organization that created it, so it can find and punish the folks who didn't donate enough. No time travel involved, no brain simulations, nothing. Actually, Roko's original version of the basilisk was a pretty nice AI that wanted to help people. A hundred people die every minute, how many of them could be saved with better technology? If donating your entire net worth would speed up the AI's creation by one day, that could be very worthwhile, according to the combined wishes of all humanity (which are encoded in the AI's utility function). Threatening to punish you is a comparatively small price to pay. Especially if you decide to comply, and the punishment never actually happens! At this point you might be indignantly asking, why would the AI decide to punish you in the future? After all, that won't help the AI in the future, because its history will already be fixed. But that doesn't matter. The AI's algorithm tries to choose the most efficient decision overall, from a "position of ignorance", rather than the most efficient decision at a particular moment in time. Technically, the algorithm tries to find the best input-output mapping according to certain criteria, not find the best output for the particular input it happened to receive. There's nothing especially futuristic about such algorithms either, you can implement one as an ordinary Python program today (for toy problems of course).

lxmorj 12y ago

So stupid. Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable. When I was a kid I watched a movie about a monster that came up from the basement when the kids in the house accidentally said an ancient Indian incantation. "BlahBlahBluey" or something I got scared that perhaps, the last thing I said "Good Night, Batman" or whatever, was in fact coincidentally an ancient curse incantation and I had summoned a monster who was en route to kill me. "Blah!" I'd say, changing my words to "Good Night, Batman Blah!" Ah, much better. But then FUCK! What if "Good Night, Batman Blah!" is the cursed incantation!!! Repeat over and over until I was sufficiently convinced that no one had ever uttered the previous sequence of sounds I had just made, become exhausted from an entirely too powerful imagination, and fall asleep. I'm pretty sure this 'thought experiment' is about as intellectual as I was at bedtime when I was 7.

teekert 12y ago
"So stupid. Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable." I think the latter is a lot more likely and I predict we'll see it before the singularity.
- probably_wrong 12y ago
  Does this machine count? http://betabeat.com/2014/06/crowdfunded-electronic-blowjob-machine-is-now-available-for-pre-order/ http://betabeat.com/2014/06/crowdfunded-electronic-blowjob-m... (warning: language mildly NSFW, plus of course a video of a blowjob machine. No naked ladies, though)
sirodoht 12y ago
> Roko's Basilisk and Roko's Infinite Blowjob Machine are equally probable. I agree. So what would they be so afraid of? My assumption is that the LessWrong community tries to come up with the best system of thinking, so by proving (by presenting a situation which shows that the best system is flawed, since we may be slaves right now) means that there really is no hope. As far as LessWrong in general, they share some controversial opinions but some of their concepts seem very interesting[1]. However, learning and especially thinking about these concepts does not appear to have any positive effect in everyday life. [1] http://lesswrong.com/lw/k5/cached_thoughts/ http://lesswrong.com/lw/k5/cached_thoughts/

iSnow 12y ago

How is this more than a souped-up version of Pascal's wager? You better believe in God now because if you don't and he exists, you'll rot in hell.

Ono-Sendai 12y ago
It sounds exactly like Pascal's wager to me, which means it suffers from all the same problems.
jarman 12y ago
Pascal's wager at least involves widely known and culturally influential concept. This is more like small kid drawing a bogeymen, being scared of it and throwing a tantrum
Tenoke 12y ago
Just in case anyone cares what the difference is - most possible Gods are about equally likely (no reason to think that one is much more likely to exist than another), while possible AIs which have reached SuperIntelligence status (to the point where they can simulate people for a start) are more likely to be certain ways (for example, self-preservation is not a consideration for an immortal God, nor is research something such a God must engage in, in order to gain knowledge, etc.).
- davidgerard 12y ago
  As far as I can tell (and I've read LW since 2010), you're asserting that on no evidence.
  - Tenoke 12y ago
    I am giving the reasons why I think so, if you disagree, you can point out what is wrong in my line of thinking: AIs are more likely to care for self-preservation (in some way), because AIs that don't are less likely to exist (since they die earlier), while Gods could have other mechanisms for survival (such as being immortal right off the bat).

madaxe_again 12y ago

It seems that everyone deriding this as pointless may be unaware of the simulation argument. It hasn't been disproven, and it's more likely than not that at some point it will be proven, given the pace of technological advancement. When you subscribe to the idea that this could all be a sim, the whole thing results in mental contortions that are literally maddening. Baudrillard observed not dissimilar in Simulacrum.

super_mario 12y ago
The cost of simulating entire universe becomes increasingly expensive to the point you need a universe of energy to simulate a universe. Daniel Dennet in his "Consciousness Explained" makes a good argument why this is so. Also there was some research done on the constraints on the universe as a numerical simulation http://arxiv.org/abs/1210.1847 http://arxiv.org/abs/1210.1847 telling us that we can do experiments to detect if we are living in a simulation (but the simulators always have a freedom to "increase the resolution" of their simulation to thwart our efforts). The paper does not discuss if there are limits on how far this game can be played on either side.
- madaxe_again 12y ago
  You don't need to simulate the entire universe in order to accurately simulate a causal chain - just the visible universe. Which is a lot smaller. Hell, you could just simulate an individual and what they observe. For all I know, I'm talking to the AI in which I, and solely I, reside right now. Woo, simulation-solipsism.

ryandvm 12y ago

I refuse to believe there are people stupid enough to fall for this tripe.

davidgerard 12y ago
You keep telling yourself that. I've had email from them. (It's not stupidity - as far as I can tell it's a combination of aspergism, OCD, a weak sense of self and reading way too much LessWrong and cut'n'pasting it into their heads. LW is a superstimulant for people of this description, which is how an organisation that really truly isn't trying to start a cult has inadvertently accumulated something around itself that is highly reminiscent of one.)

pessimizer 12y ago

Chick's Cthulhu: http://jackchick.wordpress.com/2009/07/08/chick-parody-who-will-be-eaten-first/ http://jackchick.wordpress.com/2009/07/08/chick-parody-who-w...

IgorPartola 12y ago

Really? Peter Thiel spends his time in this circle jerk? Let me give so life advice that will go unhedged, I am sure. If stuff like this gives you nightmares, get off your fucking computer. Get a different hobby so you can stop losing sleep over killer robots that you built just so they can torture you.

Tenoke 12y ago
There's about 1 confirmed person who got 'nightmares' from the basilisk (Roko), and possibly a very few others who have never said so publicly (as far as I know).
- davidgerard 12y ago
  RationalWiki editors (including me ... and Dmytry, which is part of why he can be an asshole on LW) have had email from people traumatised by the basilisk. It turns out you can't think your way out of things you did think your way into, too. This is why the RationalWiki article http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk was created, because personal email doesn't scale.
  - IgorPartola 12y ago
    So people cannot just dismiss this thing as bullshit? What is the actual thinking of these individuals? I am having a really hard time relating to them.
    - davidgerard 12y ago
      From the small sample of victims I know of, they tend to be aspergic, have OCD, have not so strong a sense of self, and buy into all the LW versions of the transhumanist memes. They realise intellectually that it's a silly idea - but something about the story pushes all their buttons, perfectly wrongly. When I stsrted getting email from them (cos I edit RW, RW was pretty much the only place on the Internet that mentioned the basilisk at the time, and LW refused to talk about it so they couldn't voice their concerns there), and found others at RW had been getting email too, I decided I had to do something to help because email wouldn't scale but it would be inhuman not to try to help. Hence the RW article on the basilisk. No email since then, so it seems to do the job. See the first talk page archive for initial discussion as we try to work out how to talk about this completely fractally stupid thing in a manner that's helpful: http://rationalwiki.org/wiki/Talk:Roko%27s_basilisk/Archive1 http://rationalwiki.org/wiki/Talk:Roko%27s_basilisk/Archive1

adwn 12y ago

> LessWrong’s founder, Eliezer Yudkowsky, is a significant figure in techno-futurism [...] What? Is this true, or just gullible journalism? > Yudkowsky said that Roko had already given nightmares to several LessWrong users and had brought them to the point of breakdown. It seems like some users of LessWrong need to spend more time in the real world and less time on stupid thought experiments.

jcbrand 12y ago

There's something very religious about the whole singularity movement. Roko's Basilisk doesn't appear very different to me than Pascal's Wager, with the added twist that the God you have to believe in is actually a malevolent demiurge (as envisioned by the Gnostic christian sects more than a 1000 years ago). If you already have a cynical viewpoint on religion, then this thought experiment is very similar to the "deal" monotheistic religions have been offering through the ages. Either believe in our god (and therefore give us, as his sole representatives enormous, unfettered power), or suffer eternal hell! (or not ;) Throughout the ages, people were terrified of hell. This is just a modern twist on an old idea. I used to be scared of hell, until I realized that hell "existed" only as long I believed in it. If I stopped believing in it, it "disappeared". That's the power of belief. Yes, it might still exist whether I believe or not, but life is full of "what ifs", the contemplation of which might drive you crazy. Better to follow a positive and compassionate path then a fear based "what if" path.

thegeomaster 12y ago

I was actually more terrified by Smile.dog :)

DalekBaldwin 12y ago

We should all just commit to believing that Box A's label actually says "Devote your life to helping create Roko's Basilisk... who will arrive in the form of the Stay Puft Marshmallow Man".