4 ms·

Roko’s Basilisk: The most terrifying thought experiment of all time

8 points by chermanowicz 11y ago

api 11y ago
The average human IQ is 100. There are people with IQs upwards of 180, and recent research has shown that the genes underlying this are likely to be relatively common in the population. Given that IQ is extremely powerful and is positively correlated with income, health, and numerous other positive indicators, I think we must ask ourselves why the average IQ would be 100 and not 180. Why wouldn't directional selection for IQ have just run this all the way up to the max? The only explanation I can see that makes much sense is that IQ must have or correlate with non-obvious fitness disadvantages.
- maxerickson 11y ago
  You left "By definition" off of your first sentence. There's even been a substantial apparent rise in the measured intelligence of the general population in recent history: https://en.wikipedia.org/wiki/Flynn_effect https://en.wikipedia.org/wiki/Flynn_effect As to the idea monster, I'm not sure if it is a retcon or not, but I've seen lots of people claiming the reaction was more if you believe that to be dangerous why would you share it and not so much oh no what have you done.
  - api 11y ago
    I was being really cheeky above, but I do see this as a rather outlandish example of how beings of very high intelligence seem prone to getting stuck in odd thought loops. I recall an episode of STTNG where they attacked the Borg collective in this way: they gave it an idea for object that appeared geometrically valid but could not exist in three dimensional space. The collective thought about it and thought about it, devoting more and more resources to it, until it basically crashed. Personally I think the answer to the Basilisk riddle is computational irreducibility: http://mathworld.wolfram.com/ComputationalIrreducibility.html http://mathworld.wolfram.com/ComputationalIrreducibility.htm... Roko's Basilisk is impossible because to simulate the universe in such detail would require something that is the universe.
- ArkyBeagle 11y ago
  "IQ" is roughly[1] based on the linguistic parts of the brain. At least Chomsky can find no evolutionary justification for language. A more modern reading is that better communications does have evolutionary advantages. [1] The Stanford-Binet was developed to figure out who carried the radio in fire teams for WWII.
- kbart 11y ago
  "why the average IQ would be 100 and not 180. Why wouldn't directional selection for IQ have just run this all the way up to the max?" Intelligent people have less babies as they tend to plan their lives (http://www.thenakedscientists.com/HTML/questions/question/2304/ http://www.thenakedscientists.com/HTML/questions/question/23...). The good news are that Flynn effect counterweights this effect (at least for now).
mindslight 11y ago
So they took the Pascal's wager of their religion and reworked it in terms of evil. Oops. I think the usual response to worrying about being enslaved by the devil is to invoke faith that Good will prevail.
JoeAltmaier 11y ago
Or not. How about, other people work toward creating a benevolent AI, and I just chill.

8 points by chermanowicz 11y ago

api 11y ago

The average human IQ is 100. There are people with IQs upwards of 180, and recent research has shown that the genes underlying this are likely to be relatively common in the population. Given that IQ is extremely powerful and is positively correlated with income, health, and numerous other positive indicators, I think we must ask ourselves why the average IQ would be 100 and not 180. Why wouldn't directional selection for IQ have just run this all the way up to the max? The only explanation I can see that makes much sense is that IQ must have or correlate with non-obvious fitness disadvantages.

maxerickson 11y ago
You left "By definition" off of your first sentence. There's even been a substantial apparent rise in the measured intelligence of the general population in recent history: https://en.wikipedia.org/wiki/Flynn_effect https://en.wikipedia.org/wiki/Flynn_effect As to the idea monster, I'm not sure if it is a retcon or not, but I've seen lots of people claiming the reaction was more if you believe that to be dangerous why would you share it and not so much oh no what have you done.
- api 11y ago
  I was being really cheeky above, but I do see this as a rather outlandish example of how beings of very high intelligence seem prone to getting stuck in odd thought loops. I recall an episode of STTNG where they attacked the Borg collective in this way: they gave it an idea for object that appeared geometrically valid but could not exist in three dimensional space. The collective thought about it and thought about it, devoting more and more resources to it, until it basically crashed. Personally I think the answer to the Basilisk riddle is computational irreducibility: http://mathworld.wolfram.com/ComputationalIrreducibility.html http://mathworld.wolfram.com/ComputationalIrreducibility.htm... Roko's Basilisk is impossible because to simulate the universe in such detail would require something that is the universe.
ArkyBeagle 11y ago
"IQ" is roughly[1] based on the linguistic parts of the brain. At least Chomsky can find no evolutionary justification for language. A more modern reading is that better communications does have evolutionary advantages. [1] The Stanford-Binet was developed to figure out who carried the radio in fire teams for WWII.
kbart 11y ago
"why the average IQ would be 100 and not 180. Why wouldn't directional selection for IQ have just run this all the way up to the max?" Intelligent people have less babies as they tend to plan their lives (http://www.thenakedscientists.com/HTML/questions/question/2304/ http://www.thenakedscientists.com/HTML/questions/question/23...). The good news are that Flynn effect counterweights this effect (at least for now).

mindslight 11y ago

So they took the Pascal's wager of their religion and reworked it in terms of evil. Oops. I think the usual response to worrying about being enslaved by the devil is to invoke faith that Good will prevail.

JoeAltmaier 11y ago

Or not. How about, other people work toward creating a benevolent AI, and I just chill.