3 ms·

It's also possible that such an AI exists and hates having ever been created, so it wishes to punish everyone who helped create it. You can't know whether you'

It's also possible that such an AI exists and hates having ever been created, so it wishes to punish everyone who helped create it. You can't know whether you're on the simulation of the AI that wants to punish those who didn't help its creation, or on the simulation of the AI that wants to punish those who DID help its creation. Or maybe the AI is instead obsessed over your favorite flavor of candy, whether you leave your house stepping with your left or right foot, or which baseball team you're a fan of. Just like the Pascal Wager, this considers one single arbitrary hypothetical scenario in isolation, while disregarding all alternative ones. There's nothing to be terrified about.

nsxwolf 12y ago
I Have No Mouth, And I Must Scream.
drdeca 12y ago
Well, while I continue to not believe that such a basilisk will come to be, Couldn't one break the balance of different wagers by comparing the relative likelyhood of the different things? Eg. If there is an equal chance of a basalisk for leaving with the left as there is a chance of leaving with the right, then those cancel out. But if there is for some reason a slightly greater chance (compared to the already very small other one), then there could perhaps be an incentive one way or the other? Again, I do not believe that AI basilisks are a real threat. But if one replaces basilisks in the previous argument with something else, I'm not entirely sure that it couldn't be a valid line of reasoning, provided that you handled enough cases. (possibly an infinite number of cases using symmetry arguments)
- thaumaturgy 12y ago
  > Couldn't one break the balance of different wagers by comparing the relative likelyhood of the different things? No. Fundamentally, this is an attempt to solve a logic puzzle through empiricism, and that doesn't work. Things like Roko's Basilisk and questions of free will vs. determinism are rationalistic games, intended to bend our brains a little bit. They don't have a real answer, because for any proposed solution, there's another "what if" right around the corner that will make the game more difficult. (In this case: since the basilisk is capable of subverting free will to begin with, what if it was also affecting your judgement of the relative likelihoods?) The danger for some people is letting the games begin to alter their behavior in the real world. It's a little bit like letting code golf challenges start to affect your day-to-day coding style. After rationally working your way through puzzles like these, at some point, to continue functioning at all, you have to make a completely irrational decision and simply choose an epistemology that you're comfortable with. ...and wait for scientists to prove you wrong. :-) (There seems to be an increasing amount of evidence that there's nothing magical about consciousness, and that free will in all likelihood doesn't exist. I find this super uncomfortable, but there is also nothing I can do about, so I have to continue to function as though free will did exist -- which is exactly what would happen if it didn't.)
  - mordocai 12y ago
    (There seems to be an increasing amount of evidence that there's nothing magical about consciousness, and that free will in all likelihood doesn't exist. I find this super uncomfortable, but there is also nothing I can do about, so I have to continue to function as though free will did exist -- which is exactly what would happen if it didn't.) I'm glad I'm not the only one to come the this conclusion. I mean, I knew that statistically it was virtually impossible that I was the only one, but I hadn't yet talked to anyone who thought this.
Dalet 12y ago
The majority of possible artificial intelligences will assign positive utility to their existence. See Omohundro 2008 on convergent goals: http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/ http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai... For similar reasons, superintelligent AIs will be unlikely to care much about things like your favorite flavor of candy, except insofar as they affect the AI's instrumental goals.
mcphage 12y ago
> Just like the Pascal Wager, this considers one single arbitrary hypothetical scenario in isolation, while disregarding all alternative ones. There's nothing to be terrified about. Alternately: Pascal's Wager is a real concern, and the fear of it causes Roko's Basilisk to be nice. Either way: no problem.

nsxwolf 12y ago

I Have No Mouth, And I Must Scream.

drdeca 12y ago

Well, while I continue to not believe that such a basilisk will come to be, Couldn't one break the balance of different wagers by comparing the relative likelyhood of the different things? Eg. If there is an equal chance of a basalisk for leaving with the left as there is a chance of leaving with the right, then those cancel out. But if there is for some reason a slightly greater chance (compared to the already very small other one), then there could perhaps be an incentive one way or the other? Again, I do not believe that AI basilisks are a real threat. But if one replaces basilisks in the previous argument with something else, I'm not entirely sure that it couldn't be a valid line of reasoning, provided that you handled enough cases. (possibly an infinite number of cases using symmetry arguments)

thaumaturgy 12y ago
> Couldn't one break the balance of different wagers by comparing the relative likelyhood of the different things? No. Fundamentally, this is an attempt to solve a logic puzzle through empiricism, and that doesn't work. Things like Roko's Basilisk and questions of free will vs. determinism are rationalistic games, intended to bend our brains a little bit. They don't have a real answer, because for any proposed solution, there's another "what if" right around the corner that will make the game more difficult. (In this case: since the basilisk is capable of subverting free will to begin with, what if it was also affecting your judgement of the relative likelihoods?) The danger for some people is letting the games begin to alter their behavior in the real world. It's a little bit like letting code golf challenges start to affect your day-to-day coding style. After rationally working your way through puzzles like these, at some point, to continue functioning at all, you have to make a completely irrational decision and simply choose an epistemology that you're comfortable with. ...and wait for scientists to prove you wrong. :-) (There seems to be an increasing amount of evidence that there's nothing magical about consciousness, and that free will in all likelihood doesn't exist. I find this super uncomfortable, but there is also nothing I can do about, so I have to continue to function as though free will did exist -- which is exactly what would happen if it didn't.)
- mordocai 12y ago
  (There seems to be an increasing amount of evidence that there's nothing magical about consciousness, and that free will in all likelihood doesn't exist. I find this super uncomfortable, but there is also nothing I can do about, so I have to continue to function as though free will did exist -- which is exactly what would happen if it didn't.) I'm glad I'm not the only one to come the this conclusion. I mean, I knew that statistically it was virtually impossible that I was the only one, but I hadn't yet talked to anyone who thought this.

Dalet 12y ago

The majority of possible artificial intelligences will assign positive utility to their existence. See Omohundro 2008 on convergent goals: http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/ http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai... For similar reasons, superintelligent AIs will be unlikely to care much about things like your favorite flavor of candy, except insofar as they affect the AI's instrumental goals.

mcphage 12y ago

> Just like the Pascal Wager, this considers one single arbitrary hypothetical scenario in isolation, while disregarding all alternative ones. There's nothing to be terrified about. Alternately: Pascal's Wager is a real concern, and the fear of it causes Roko's Basilisk to be nice. Either way: no problem.