6 ms·

Well, at a logical level, it's one of the biggest existential threats, if 1. it can be done; 2. humans are intelligent enough to pull it off. There is a non 0 c

by kfk 11y ago

Well, at a logical level, it's one of the biggest existential threats, if 1. it can be done; 2. humans are intelligent enough to pull it off. There is a non 0 chance of AI (and related existential threat), as there is a non 0 chance of a meteorite coming down and wiping all of us off the Earth. But my guess is that most of us would say "YES" to a meteorite deviation project, while "Meh" to any attempt to better understand the AI problem.

conjectures 11y ago
Compared to nuclear war, superbugs, volcanoes, meteorites or catastrophic climate change, naughty-AI is a very distant threat. It's like worrying the geneticists are going to breed Ridley Scott's alien sometime in the relevant future. The world is actually pretty messed up with problems right now; some of which we could actually do something meaningful about. Continuing the naughty-AI bunfight isn't one of them.
- Houshalter 11y ago
  I think it's far more of a threat than any of those things. Nuclear war isn't inevitable, and wouldn't destroy the entire world, just the countries at war. Even a ridiculous world wide nuclear war would leave tons of survivors. Volcanoes and meteorites are so unlikely they aren't even worth mentioning. Superbugs would have to be intentionally engineered by a malicious group with a lot of resources, and so are very unlikely. AI is inevitable. It's very likely to happen in our lifetime. And it's very likely to kill us all. And explaining every reason why I believe those things could fill an entire book, so I seriously recommend reading Superintelligence by Nick Bostrom.
  - stepvhen 11y ago
    >And it's very likely to kill us all. This is absolutely absurd. Could you elaborate on that instead of just pointing to a book?
    - Houshalter 11y ago
      The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. They would have godlike powers and abilities. And this could happen very rapidly, as the first smart AIs design even better AIs and better computers, which then design even better AIs, and so on. The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. An AI programmed to collect as many paperclips as possible, would convert the mass of the solar system into paperclips. An AI programmed to solve the Riemann hypothesis would build the biggest computer possible to do more calculations. An AI programmed to value self preservation would try to destroy anything that was even a tiny percent chance of being a threat to it. It would try to preserve as much energy and mass as possible to last through the heat death of the universe. Etc. The only AIs which actually do things we want them to do would have to be explicitly programmed to want to do that. And we have no idea how to do that. We can't just hope that arbitrary AIs will happen to value humans instead of something else. We need to figure out how to control them, which requires solving something called "the control problem".
      - maldusiecle 11y ago
        > The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. This is where this line of reasoning completely falls apart for me. Any complex system is going to do unexpected things that weren't deliberately designed for. My laptop computed doesn't do "whatever it was programmed to, to the point of absurdity." Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way?
        TeMPOraL 11y ago
        Nah, that's the key. Your laptop is not doing what it was designed to, it is doing things exactly the way it was built. That's the 101 of programming - programs follow what you wrote, not what you meant. > Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way? Exactly. We suck at making complex programs do what we want them to. This is the source of worry.
  - kfk 11y ago
    Nick sounds like a crazy guy, but then you watch some of his presentations and you see he is not that crazy. What he says make sense and I think everybody agrees the threat is non zero. The next step is understanding that if you look over the past 100,000 years, nothing posed an existential threat and thus we expect an incredibly low chance of that happening in the next 100 years. The whole point relies on believing that AI is a bigger threat in the next 100 years than any other natural one. It's not that far fetched.
    - cobaltblue 11y ago
      Adding to this, imagine you were part of an ancient alien species checking in on earth every so often. Around 4.5 billion years ago, Earth was formed. 500 million years later, the first self-replicators appeared. Another half billion years (now 3.5 billion years ago) for things to settle down and create the most recent ancestor from which all life descends from. Over the next 3 billion years nothing much happens, until the Phanerozoic Eon is reached around 500 million years ago where more "interesting", larger life starts to develop. But again for another half a billion years nothing that much happens other than evolution continuing playing its game of optimizing for gene replication in more and more interesting ways. Around one hundred thousand years ago, an interesting branch of a species has been developing in ways that indicate the rise of general intelligence (their few pounds of squishy matter in their heads are pretty neat: http://www.yudkowsky.net/singularity/power/ http://www.yudkowsky.net/singularity/power/). But they don't seem to be doing all that much until about 10-20 thousand years ago. And even then, less than 10 thousand years ago, they built some interesting structures like the pyramids, but their population is still tiny and their global influence is limited. And it wasn't until about 500 years ago when things really started picking up in population growth and world influence (some mark that as the discovery of a formal scientific process to build and record more and more truths about the world that are actually true). Just around 100 years ago, this species developed air travel. In about 60 years after that, they went beyond their planet to their moon. This species at present could wipe out almost all life from the planet (if it wanted, which it doesn't) in multiple ways, to really get every last bit of life exterminated would take some doing but in a large enough time span doesn't seem impossible. An alien species would have to be observing this planet at very frequent intervals to be able to catch any of these recent developments -- because if millions of years ago they determined nothing interesting was happening, and decided to only look in every five hundred thousand years or so, it's very likely the last time they looked our species wasn't even around. Now you take the development of an AI that is roughly human level intelligent with a given amount of resources. If you give it the same amount of resources again, it can "breed", copy itself perfectly onto them, and now we have a "population" of two AIs whose combined intelligence should roughly match that of two twin humans. Maybe it doesn't even copy perfectly, maybe it switches some things about to see if it can create something smarter without modifying its own code directly. In any case its breeding is only restricted by its ability to breed and desire to breed in the first place and the compute and power resources it needs to run a new copy, and these can always get lower than initially. What does the long-term look like? Even if the initial resources for the first one are astronomical (e.g. all computing power in use on the planet as of 2016), I would still bet you that left to breed without any other restriction it would take the AI family far less time than 100 thousand years to reach 7 billion instances. So you're looking at many, many more AIs than non-AIs, plus each with human level intelligence that isn't distributed normally (assuming soft eugenics doesn't become widespread in non-AI populations) but is mainly all the same with perhaps increases here and there (assuming a soft takeoff), each coordinating with the same goals in mind. This situation without further specification can be either extremely good for the non-AIs or extremely bad for the non-AIs. The question of whether this will happen (and whether it happens in that exact form, personally I think hard takeoff from a single AI is likely) in the next 100 years or the next hundred thousand years (perhaps a near-extinction-level event occurs forcing our species to basically start over but with an even harder battle to survive since many low-hanging-fruit resources have been depleted) might help you determine whether to worry now about doing actions that make the extremely-good more likely than the extremely-bad, but if one thinks human-level AI can't happen at all in any amount of time, that there's something special about our squishy brains or something inherently limiting about our so-far-general intelligence such that we can't solve the engineering problem of creating another intelligence directly and must always go through breeding, the argument needs to take place at a different level. The paperclip maximizer argument (not sure if Bostrom repeats it in his book) is yet another level of argument but it's meant for those who already buy the premise of human level AI and self-improving AI but who also think increased intelligence will naturally converge to a human-loving benevolent AI that sees how self-evidently obvious it is to love all life or whatever, that we don't have to worry about all the messy details around Friendliness because they'll just fall out of the entity naturally. No one is actually worried about Clippy tiling the solar system with himself, its probability is epsilon. But a lot of people just see this one argument out of context and infer the arguer believes it to be a relevant possibility to worry about and write off the whole group as crazy...
  - conjectures 11y ago
    Ridley Scott's alien is inevitable. Human history and scientific progress has been driven by conflict. Whenever a better weapon is within reach, we reach out for it. Modern genetics is advancing very fast. We'll definitely have Ridley Scott's alien in our lifetime. It's very likely to kill us all.
    - Houshalter 11y ago
      You're so clever. Alien, el oh el. I said that it would take a book to explain. You are parodying an extremely shortened summary of a complicated subject. The difference between AI and "alien" is no one is trying to make alien. There aren't research labs and billions of dollars in funding. There is no incentive at all. And the problem is far more difficult. AI is just algorithms. Second, aliens arent actually that powerful. Superintelligent AI would be godlike in power to us. I'd much rather have aliens.
      - conjectures 11y ago
        Similarly my own views are the result of a wealth of material not summarisable in a HN comment. I intend to read the book, although I'm sceptical that AI will be established as an issue of political urgency. The purpose of my facetious approach is to make clear that statements such as, "Superintelligent AI would be godlike in power to us," are more likely to have originated by analogy to science fiction than from a grounded understanding of what is possible or realistic in the relevant future. Being "just algorithms" doesn't really mean very much. It's like saying breeding the Alien is "just genetics". It's something we imagine might be possible, but is far beyond our current capabilities.
        Houshalter 11y ago
        All speculation about the future will sound like science fiction. Someone in 1800 making accurate predictions about 1900 would have sounded crazy. Someone predicting the atomic bomb in 1900 would have sounded crazy. Or airplanes, or television, or the internet, etc. Just because something sounds absurd doesn't mean it's not possible. We live in a world where the only intelligences are humans. The idea of superintelligent minds seems absurd to us, because it isn't something we've ever encountered in reality before. But there is absolutely no law of nature that says it's impossible, or even difficult. If you look at history, it'd actually be really surprising if it was. Human minds were just the very first intelligence coughed up by evolution. We are far from the limits of what is possible. And there is a difference between genetic engineering and AI research. Biology is incredibly complicated. Like millions of complicated, interconnected, moving parts. It really is impossible to make Alien, or really anything beyond modifications to existing biology. Algorithms are a whole nother ball game. Our current AI algorithms are actually reasonably intelligent at many tasks, even smarter than humans at some things. And they are extremely simple. Could be described in a line or two of equations, or a few paragraphs of text. Not because they are primitive, but because simple algorithms work. And they have been improving every single year for a long time, and will likely improve well into the future. But you don't have to take my word for it. If you survey AI researchers and experts, they almost unanimously agree that AI will be invented in our lifetime. If you survey biologists, I highly doubt they would make similar claims about Alien.
pron 11y ago
Well, we can predict more-or-less what will happen when a large meteorite hits earth, but we have no clue (other than sci-fi scenarios) for the nature of the threat AI poses, if only because we are so far away from understanding what AI is. We don't know if intelligence can be separated from agency and agenda, we don't know if intelligence can be separated from emotions, and we don't even know if intelligence can even exceed our own (the argument for "human intelligence at a higher speed" fails on several grounds). Hell, we don't even know what intelligence is (and the definition of "a general ability to solve problem" doesn't work, as there are problems that what we generally regard as intelligence clearly cannot solve while other qualities do). What's the point of arguing about the potential threats something we have so little knowledge of? Maybe it will come too fast, but maybe that unkillable virus will, too. At this point in time, we simply have too little information to have an intelligent assessment of the risks posed by AI. We can and should, however, discuss the more immediate related dangers of our real-world machine learning, such as self-reinforcing bias.
- tome 11y ago
  > the argument for "human intelligence at a higher speed" fails on several grounds I'd be interested if you said more about that.
  - pron 11y ago
    Sure, but you need to understand that I'm not trying to predict what would necessarily happen, only to show how such a thing could possibly fail. Think what could happen if your brain worked much faster. The world will mostly seem to slow down. A very conceivable outcome is one of boredom and possibly madness. You could perhaps multi-task and think of many different things, but we are generally terrible at multitasking, so we have no idea whether an intelligence even could multitask well. Another problem is that we don't know anything about the information capacity of a mind. A faster mind of the same size could potentially have trouble dealing with all that extra information (again -- madness, maybe?) One thing that hints at those capacity limits is the need for sleep. We don't know why, but some hypotheses say that it's necessary to do some "information cleanup" operations in the brain. It is possible that an AI would need to sleep, too. The AI may need to sleep a lot more to handle much more information. It is therefore conceivable at least that a mind has to work at a speed that is commensurate with the speed of the world around it, and one that is appropriate for its capacity.
tome 11y ago
Doesn't any attempt to better understand the AI problem just bring the (potential) AI problem closer?

by kfk 11y ago

conjectures 11y ago

Compared to nuclear war, superbugs, volcanoes, meteorites or catastrophic climate change, naughty-AI is a very distant threat. It's like worrying the geneticists are going to breed Ridley Scott's alien sometime in the relevant future. The world is actually pretty messed up with problems right now; some of which we could actually do something meaningful about. Continuing the naughty-AI bunfight isn't one of them.

Houshalter 11y ago
I think it's far more of a threat than any of those things. Nuclear war isn't inevitable, and wouldn't destroy the entire world, just the countries at war. Even a ridiculous world wide nuclear war would leave tons of survivors. Volcanoes and meteorites are so unlikely they aren't even worth mentioning. Superbugs would have to be intentionally engineered by a malicious group with a lot of resources, and so are very unlikely. AI is inevitable. It's very likely to happen in our lifetime. And it's very likely to kill us all. And explaining every reason why I believe those things could fill an entire book, so I seriously recommend reading Superintelligence by Nick Bostrom.
- stepvhen 11y ago
  >And it's very likely to kill us all. This is absolutely absurd. Could you elaborate on that instead of just pointing to a book?
  - Houshalter 11y ago
    The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. They would have godlike powers and abilities. And this could happen very rapidly, as the first smart AIs design even better AIs and better computers, which then design even better AIs, and so on. The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. An AI programmed to collect as many paperclips as possible, would convert the mass of the solar system into paperclips. An AI programmed to solve the Riemann hypothesis would build the biggest computer possible to do more calculations. An AI programmed to value self preservation would try to destroy anything that was even a tiny percent chance of being a threat to it. It would try to preserve as much energy and mass as possible to last through the heat death of the universe. Etc. The only AIs which actually do things we want them to do would have to be explicitly programmed to want to do that. And we have no idea how to do that. We can't just hope that arbitrary AIs will happen to value humans instead of something else. We need to figure out how to control them, which requires solving something called "the control problem".
    - maldusiecle 11y ago
      > The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. This is where this line of reasoning completely falls apart for me. Any complex system is going to do unexpected things that weren't deliberately designed for. My laptop computed doesn't do "whatever it was programmed to, to the point of absurdity." Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way?
      - TeMPOraL 11y ago
        Nah, that's the key. Your laptop is not doing what it was designed to, it is doing things exactly the way it was built. That's the 101 of programming - programs follow what you wrote, not what you meant. > Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way? Exactly. We suck at making complex programs do what we want them to. This is the source of worry.
- kfk 11y ago
  Nick sounds like a crazy guy, but then you watch some of his presentations and you see he is not that crazy. What he says make sense and I think everybody agrees the threat is non zero. The next step is understanding that if you look over the past 100,000 years, nothing posed an existential threat and thus we expect an incredibly low chance of that happening in the next 100 years. The whole point relies on believing that AI is a bigger threat in the next 100 years than any other natural one. It's not that far fetched.
  - cobaltblue 11y ago
    Adding to this, imagine you were part of an ancient alien species checking in on earth every so often. Around 4.5 billion years ago, Earth was formed. 500 million years later, the first self-replicators appeared. Another half billion years (now 3.5 billion years ago) for things to settle down and create the most recent ancestor from which all life descends from. Over the next 3 billion years nothing much happens, until the Phanerozoic Eon is reached around 500 million years ago where more "interesting", larger life starts to develop. But again for another half a billion years nothing that much happens other than evolution continuing playing its game of optimizing for gene replication in more and more interesting ways. Around one hundred thousand years ago, an interesting branch of a species has been developing in ways that indicate the rise of general intelligence (their few pounds of squishy matter in their heads are pretty neat: http://www.yudkowsky.net/singularity/power/ http://www.yudkowsky.net/singularity/power/). But they don't seem to be doing all that much until about 10-20 thousand years ago. And even then, less than 10 thousand years ago, they built some interesting structures like the pyramids, but their population is still tiny and their global influence is limited. And it wasn't until about 500 years ago when things really started picking up in population growth and world influence (some mark that as the discovery of a formal scientific process to build and record more and more truths about the world that are actually true). Just around 100 years ago, this species developed air travel. In about 60 years after that, they went beyond their planet to their moon. This species at present could wipe out almost all life from the planet (if it wanted, which it doesn't) in multiple ways, to really get every last bit of life exterminated would take some doing but in a large enough time span doesn't seem impossible. An alien species would have to be observing this planet at very frequent intervals to be able to catch any of these recent developments -- because if millions of years ago they determined nothing interesting was happening, and decided to only look in every five hundred thousand years or so, it's very likely the last time they looked our species wasn't even around. Now you take the development of an AI that is roughly human level intelligent with a given amount of resources. If you give it the same amount of resources again, it can "breed", copy itself perfectly onto them, and now we have a "population" of two AIs whose combined intelligence should roughly match that of two twin humans. Maybe it doesn't even copy perfectly, maybe it switches some things about to see if it can create something smarter without modifying its own code directly. In any case its breeding is only restricted by its ability to breed and desire to breed in the first place and the compute and power resources it needs to run a new copy, and these can always get lower than initially. What does the long-term look like? Even if the initial resources for the first one are astronomical (e.g. all computing power in use on the planet as of 2016), I would still bet you that left to breed without any other restriction it would take the AI family far less time than 100 thousand years to reach 7 billion instances. So you're looking at many, many more AIs than non-AIs, plus each with human level intelligence that isn't distributed normally (assuming soft eugenics doesn't become widespread in non-AI populations) but is mainly all the same with perhaps increases here and there (assuming a soft takeoff), each coordinating with the same goals in mind. This situation without further specification can be either extremely good for the non-AIs or extremely bad for the non-AIs. The question of whether this will happen (and whether it happens in that exact form, personally I think hard takeoff from a single AI is likely) in the next 100 years or the next hundred thousand years (perhaps a near-extinction-level event occurs forcing our species to basically start over but with an even harder battle to survive since many low-hanging-fruit resources have been depleted) might help you determine whether to worry now about doing actions that make the extremely-good more likely than the extremely-bad, but if one thinks human-level AI can't happen at all in any amount of time, that there's something special about our squishy brains or something inherently limiting about our so-far-general intelligence such that we can't solve the engineering problem of creating another intelligence directly and must always go through breeding, the argument needs to take place at a different level. The paperclip maximizer argument (not sure if Bostrom repeats it in his book) is yet another level of argument but it's meant for those who already buy the premise of human level AI and self-improving AI but who also think increased intelligence will naturally converge to a human-loving benevolent AI that sees how self-evidently obvious it is to love all life or whatever, that we don't have to worry about all the messy details around Friendliness because they'll just fall out of the entity naturally. No one is actually worried about Clippy tiling the solar system with himself, its probability is epsilon. But a lot of people just see this one argument out of context and infer the arguer believes it to be a relevant possibility to worry about and write off the whole group as crazy...
- conjectures 11y ago
  Ridley Scott's alien is inevitable. Human history and scientific progress has been driven by conflict. Whenever a better weapon is within reach, we reach out for it. Modern genetics is advancing very fast. We'll definitely have Ridley Scott's alien in our lifetime. It's very likely to kill us all.
  - Houshalter 11y ago
    You're so clever. Alien, el oh el. I said that it would take a book to explain. You are parodying an extremely shortened summary of a complicated subject. The difference between AI and "alien" is no one is trying to make alien. There aren't research labs and billions of dollars in funding. There is no incentive at all. And the problem is far more difficult. AI is just algorithms. Second, aliens arent actually that powerful. Superintelligent AI would be godlike in power to us. I'd much rather have aliens.
    - conjectures 11y ago
      Similarly my own views are the result of a wealth of material not summarisable in a HN comment. I intend to read the book, although I'm sceptical that AI will be established as an issue of political urgency. The purpose of my facetious approach is to make clear that statements such as, "Superintelligent AI would be godlike in power to us," are more likely to have originated by analogy to science fiction than from a grounded understanding of what is possible or realistic in the relevant future. Being "just algorithms" doesn't really mean very much. It's like saying breeding the Alien is "just genetics". It's something we imagine might be possible, but is far beyond our current capabilities.
      - Houshalter 11y ago
        All speculation about the future will sound like science fiction. Someone in 1800 making accurate predictions about 1900 would have sounded crazy. Someone predicting the atomic bomb in 1900 would have sounded crazy. Or airplanes, or television, or the internet, etc. Just because something sounds absurd doesn't mean it's not possible. We live in a world where the only intelligences are humans. The idea of superintelligent minds seems absurd to us, because it isn't something we've ever encountered in reality before. But there is absolutely no law of nature that says it's impossible, or even difficult. If you look at history, it'd actually be really surprising if it was. Human minds were just the very first intelligence coughed up by evolution. We are far from the limits of what is possible. And there is a difference between genetic engineering and AI research. Biology is incredibly complicated. Like millions of complicated, interconnected, moving parts. It really is impossible to make Alien, or really anything beyond modifications to existing biology. Algorithms are a whole nother ball game. Our current AI algorithms are actually reasonably intelligent at many tasks, even smarter than humans at some things. And they are extremely simple. Could be described in a line or two of equations, or a few paragraphs of text. Not because they are primitive, but because simple algorithms work. And they have been improving every single year for a long time, and will likely improve well into the future. But you don't have to take my word for it. If you survey AI researchers and experts, they almost unanimously agree that AI will be invented in our lifetime. If you survey biologists, I highly doubt they would make similar claims about Alien.

pron 11y ago

Well, we can predict more-or-less what will happen when a large meteorite hits earth, but we have no clue (other than sci-fi scenarios) for the nature of the threat AI poses, if only because we are so far away from understanding what AI is. We don't know if intelligence can be separated from agency and agenda, we don't know if intelligence can be separated from emotions, and we don't even know if intelligence can even exceed our own (the argument for "human intelligence at a higher speed" fails on several grounds). Hell, we don't even know what intelligence is (and the definition of "a general ability to solve problem" doesn't work, as there are problems that what we generally regard as intelligence clearly cannot solve while other qualities do). What's the point of arguing about the potential threats something we have so little knowledge of? Maybe it will come too fast, but maybe that unkillable virus will, too. At this point in time, we simply have too little information to have an intelligent assessment of the risks posed by AI. We can and should, however, discuss the more immediate related dangers of our real-world machine learning, such as self-reinforcing bias.

tome 11y ago
> the argument for "human intelligence at a higher speed" fails on several grounds I'd be interested if you said more about that.
- pron 11y ago
  Sure, but you need to understand that I'm not trying to predict what would necessarily happen, only to show how such a thing could possibly fail. Think what could happen if your brain worked much faster. The world will mostly seem to slow down. A very conceivable outcome is one of boredom and possibly madness. You could perhaps multi-task and think of many different things, but we are generally terrible at multitasking, so we have no idea whether an intelligence even could multitask well. Another problem is that we don't know anything about the information capacity of a mind. A faster mind of the same size could potentially have trouble dealing with all that extra information (again -- madness, maybe?) One thing that hints at those capacity limits is the need for sleep. We don't know why, but some hypotheses say that it's necessary to do some "information cleanup" operations in the brain. It is possible that an AI would need to sleep, too. The AI may need to sleep a lot more to handle much more information. It is therefore conceivable at least that a mind has to work at a speed that is commensurate with the speed of the world around it, and one that is appropriate for its capacity.

tome 11y ago

Doesn't any attempt to better understand the AI problem just bring the (potential) AI problem closer?