8 ms·

Apocalypse soon: the scientists preparing for the end times

42 points by HillRat 12y ago

chroma 12y ago
Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying to make. If general AI research is successful, we'll need to build agents with some very carefully chosen goals. Even something as silly as "make paperclips" can result in a universe tiled with paperclips. Earth and its biomass could be turned into said paperclips, ending humanity.
- mariusz79 12y ago
  I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if it does not take under consideration that destroying humans to make paper clips does not make sense.
  - OMFGscience 12y ago
    Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?
    - mariusz79 12y ago
      I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
      - TeMPOraL 12y ago
        > I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Given [0], I think that it's the only plausible way we can have to make it follow human values. > True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1]. > I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed. Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious. [0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ [1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis
      - aaronem 12y ago
        The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously. The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
  - DanAndersen 12y ago
    One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.
    - mariusz79 12y ago
      >Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
      - hawleyal 12y ago
        The point still stands. I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting. The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
      - mariusz79 12y ago
        @hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence. My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless. I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff. It's nothing but a fear-porn designed to get more funding.
  - TeMPOraL 12y ago
    I see people down this discussion thread are doing one thing wrong - you guys are anthropomorphizing intelligence. Paperclip AI may be stupid by human standards, but is a perfectly valid optimization process. When people talk about AIs, they don't mean intelligence as "thinking like humans would think"; they mean a very powerful optimization process. Optimization processes have no common notion of "stupidity", especially not one matching what humans call stupid. They just optimize for whatever they do. Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized. Keep that in mind before saying that "AI wouldn't be that stupid to do this". http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/ http://lesswrong.com/lw/vb/efficient_crossdomain_optimizatio...
  - wavegeek 12y ago
    For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.
    - mariusz79 12y ago
      no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.
      - JamesArgo 12y ago
        The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal. >If you have to specify the goals very carefully then the AI it's not super intelligent by definition. When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
    - sitkack 12y ago
      By analogy the AI is a leprechaun or a genie that will give you what you asked for but in a way that could do you harm.
- deleted 12y ago
  [deleted]
- Zigurd 12y ago
  > Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. He gets halfway to right. Recently there was a thread about suicide on HN. Nearly all the comments on that thread ignored that humans, like other animals, are built to propagate the information in their genes, and argued against suicide based on concepts that are driven by or that rationalize subconscious drives. An AI, even one with a programmed goal of propagating itself, does not carry information in genes and the information in an AI was never selected for the actual ability to propagate in a larger system that thinks it is in control of its goals. Real AIs are likely to be extremely alien. Even in warning people about anthropomorpizing, there's a lot of anthropomorphism. Evil AIs are depicted as human evil. So much of human motivation is closed to introspection that imagining alien and evil motivation is unlikely to be close to the eventual reality. This probably won't be widely understood until AIs are built. And what if they all choose to erase themsleves?
- api 12y ago
  I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"
  - JulianMorrison 12y ago
    You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.
    - mariusz79 12y ago
      I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?
      - JamesArgo 12y ago
        You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function. >But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent? That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.
        mariusz79 12y ago
dosh 12y ago
It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?
- adrianN 12y ago
  Gradual loss of skills and knowledge leading to a collapse of technological society.
- Havvy 12y ago
  Anarchy won't cause an apocalypse, just a change in the structure of the ruling class as a new ruling structure is created. also on the extra terrestrial influence side, there's direct hits by gamma rays that could wipe out Earth in an instant.
  - MoistDinosaur 12y ago
    Anarchy causing a collapse of the ruling structure could create a war that destroys much of society. Even without anarchy, imagine a country like China or the US plunged into a civil war. The effects on the global economy would be disastrous
- seren 12y ago
  Solar storm destroying a large part of the grid, leading to some heavy disruption including nuclear meltdown. The inversion of magnetic pole could have some impact on how we are shielded from space radiation.
- TeMPOraL 12y ago
  Here you go: http://en.wikipedia.org/wiki/Global_catastrophic_risks http://en.wikipedia.org/wiki/Global_catastrophic_risks :). The keywords are: existential risks.
- Aardwolf 12y ago
aaronem 12y ago
> “Being born into a moment when the fate of the universe is at stake is a lot of fun.” Hubris, like heroin, offers an approximation of ecstasy. Therein lies its danger.
bob_situmorang 12y ago
it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly.

42 points by HillRat 12y ago

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying to make. If general AI research is successful, we'll need to build agents with some very carefully chosen goals. Even something as silly as "make paperclips" can result in a universe tiled with paperclips. Earth and its biomass could be turned into said paperclips, ending humanity.

mariusz79 12y ago
I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if it does not take under consideration that destroying humans to make paper clips does not make sense.
- OMFGscience 12y ago
  Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?
  - mariusz79 12y ago
    I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
    - TeMPOraL 12y ago
      > I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Given [0], I think that it's the only plausible way we can have to make it follow human values. > True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1]. > I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed. Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious. [0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ [1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis
    - aaronem 12y ago
      The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously. The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
- DanAndersen 12y ago
  One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.
  - mariusz79 12y ago
    >Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
    - hawleyal 12y ago
      The point still stands. I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting. The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
    - mariusz79 12y ago
      @hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence. My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless. I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff. It's nothing but a fear-porn designed to get more funding.
- TeMPOraL 12y ago
  I see people down this discussion thread are doing one thing wrong - you guys are anthropomorphizing intelligence. Paperclip AI may be stupid by human standards, but is a perfectly valid optimization process. When people talk about AIs, they don't mean intelligence as "thinking like humans would think"; they mean a very powerful optimization process. Optimization processes have no common notion of "stupidity", especially not one matching what humans call stupid. They just optimize for whatever they do. Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized. Keep that in mind before saying that "AI wouldn't be that stupid to do this". http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/ http://lesswrong.com/lw/vb/efficient_crossdomain_optimizatio...
- wavegeek 12y ago
  For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.
  - mariusz79 12y ago
    no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.
    - JamesArgo 12y ago
      The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal. >If you have to specify the goals very carefully then the AI it's not super intelligent by definition. When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
  - sitkack 12y ago
    By analogy the AI is a leprechaun or a genie that will give you what you asked for but in a way that could do you harm.
deleted 12y ago
[deleted]
Zigurd 12y ago
> Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. He gets halfway to right. Recently there was a thread about suicide on HN. Nearly all the comments on that thread ignored that humans, like other animals, are built to propagate the information in their genes, and argued against suicide based on concepts that are driven by or that rationalize subconscious drives. An AI, even one with a programmed goal of propagating itself, does not carry information in genes and the information in an AI was never selected for the actual ability to propagate in a larger system that thinks it is in control of its goals. Real AIs are likely to be extremely alien. Even in warning people about anthropomorpizing, there's a lot of anthropomorphism. Evil AIs are depicted as human evil. So much of human motivation is closed to introspection that imagining alien and evil motivation is unlikely to be close to the eventual reality. This probably won't be widely understood until AIs are built. And what if they all choose to erase themsleves?
api 12y ago
I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"
- JulianMorrison 12y ago
  You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.
  - mariusz79 12y ago
    I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?
    - JamesArgo 12y ago
      You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function. >But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent? That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.
      - mariusz79 12y ago

dosh 12y ago

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

adrianN 12y ago
Gradual loss of skills and knowledge leading to a collapse of technological society.
Havvy 12y ago
Anarchy won't cause an apocalypse, just a change in the structure of the ruling class as a new ruling structure is created. also on the extra terrestrial influence side, there's direct hits by gamma rays that could wipe out Earth in an instant.
- MoistDinosaur 12y ago
  Anarchy causing a collapse of the ruling structure could create a war that destroys much of society. Even without anarchy, imagine a country like China or the US plunged into a civil war. The effects on the global economy would be disastrous
seren 12y ago
Solar storm destroying a large part of the grid, leading to some heavy disruption including nuclear meltdown. The inversion of magnetic pole could have some impact on how we are shielded from space radiation.
TeMPOraL 12y ago
Here you go: http://en.wikipedia.org/wiki/Global_catastrophic_risks http://en.wikipedia.org/wiki/Global_catastrophic_risks :). The keywords are: existential risks.
Aardwolf 12y ago

aaronem 12y ago

> “Being born into a moment when the fate of the universe is at stake is a lot of fun.” Hubris, like heroin, offers an approximation of ecstasy. Therein lies its danger.

bob_situmorang 12y ago

it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly.