5 ms·

>And it's very likely to kill us all. This is absolutely absurd. Could you elaborate on that instead of just pointing to a book?

Houshalter 11y ago
The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. They would have godlike powers and abilities. And this could happen very rapidly, as the first smart AIs design even better AIs and better computers, which then design even better AIs, and so on. The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. An AI programmed to collect as many paperclips as possible, would convert the mass of the solar system into paperclips. An AI programmed to solve the Riemann hypothesis would build the biggest computer possible to do more calculations. An AI programmed to value self preservation would try to destroy anything that was even a tiny percent chance of being a threat to it. It would try to preserve as much energy and mass as possible to last through the heat death of the universe. Etc. The only AIs which actually do things we want them to do would have to be explicitly programmed to want to do that. And we have no idea how to do that. We can't just hope that arbitrary AIs will happen to value humans instead of something else. We need to figure out how to control them, which requires solving something called "the control problem".
- maldusiecle 11y ago
  > The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. This is where this line of reasoning completely falls apart for me. Any complex system is going to do unexpected things that weren't deliberately designed for. My laptop computed doesn't do "whatever it was programmed to, to the point of absurdity." Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way?
  - TeMPOraL 11y ago
    Nah, that's the key. Your laptop is not doing what it was designed to, it is doing things exactly the way it was built. That's the 101 of programming - programs follow what you wrote, not what you meant. > Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way? Exactly. We suck at making complex programs do what we want them to. This is the source of worry.
  - Houshalter 11y ago
    Your laptop computer does do what it was programmed to do, to the point of absurdity. It will keep executing a computer bug over and ever. Despite the fact that isn't what you intended, it still executes the code anyway. An AI will continue to execute, even when it stops doing what we intended it to do. Computers don't care about our original intentions, they just execute code. A program "complex enough to simulate the human mind" would probably have even more bugs that result in unintentional behavior. Not fewer.
- stepvhen 11y ago
  >The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. >The second point is that they would have no reason to keep us around. Questions: (1) What reason do we have to keep chimpanzee's around? Why don't we just kill them all? They can't even do calculus. (2) Whatever the answer for question 1, wouldn't an AI consider that same scenario regarding humans? Assuming an AI is truly smarter than a human? (3) If AI gets to the point where they are much smarter than humans, wouldn't it follow that there would be AI philosophers? Or activists? Or pacifists? Would there be such a thing as an AI civilization? Or is your scenario just a few AI overlords that some how slipped through the cracks of regulation by humans?
  - TeMPOraL 11y ago
    1. Because we have empathy and because we have no compelling economic reason to (economics always overruling silly things like empathy). 2. Empathy, which is a part of our emotions, is an incredibly complex thing we can't even begin to formalize yet - it's extremely unlikely that a random AI we'd build would happen to have the exact same emotional makeup as we do. 3. "Smart" has a particular meaning here when used in discussing AI. It implies raw intelligence, not morals. A smart AI may be incredibly good at evaluating information, inferring new facts and doing complex tasks. None of it means it will necessarily have to ask itself philosophical questions that we do - those actually mostly come from our emotions, from our desire to understand our place in the universe. The core of the AI-threat issue is that for an AI to not be dangerous it would have to value us and the same things we value. We can't agree about our exact values even among ourselves, and we're nowhere near even trying to formalize it on a level of detail that would be useful for programming.
    - mcguire 11y ago
      That's a good point. How is AI worse than a smart, amoral human?
      - ddingus 11y ago
        It may have very long time frames to work in, may be faster or able to evade the nefarious detection intent of humans by working in ways very different from how we work. Read a book recently that contained an interesting AI attack. Basically, it set a lot of small objects in motion, with complex orbits all computed to result in a large impact at some point in the future. We would not notice something done this way, by analogy.
- mcguire 11y ago
  I hate to be the logical positivist here (I really, really do), but I'm not sure what your first two sentences mean. What would "smarter than humans" mean and how exactly is it going to happen?
  - CuriousSkeptic 11y ago
    My take on how it's going to happen is that it allready has. Most technologies we build, and use, surpass humans in some capacity. Thinking is no exceptions and we allready have machines far better than humans at some thnking tasks, such as arithmetic. We are now working on systems surpassing humans in various recognition and prediction tasks. I suspect the catalog of tasks far better done by machines will continue to expand to finally encompass all tasks required to form a general intelligence. Only, the trick to assemble them, into a complete system could very well prove tricky. The result, though, is that there will never exist such a thing as human level artificial intelligence because, if and when, a system of general intelligence is created all its components will far surpass human equivalent capabilities. So "smarter" simply means far superior to any human in any task requiring thinking, in the same sense that computers are far superior to any human at arithmetic. Even some odd genius savant of today cannot compete with computers at number crunching. More interestingly it's not even interesting to compare, the difference in potential performance is just at different scales all together.
cobaltblue 11y ago
I think a shorter tl;dr is simply that the ways a human-level or beyond AI can be programmed incorrectly (it eventually kills us all or worse), even trying our best to do it right, outnumber the ways it can be programmed correctly. If you don't believe this, I'm not sure how to convince you, other than to poke holes in your ideas about how a benevolent AI could be programmed correctly so you see that such an idea is actually a terrible one or at least insufficient. There are many level of detail attacks I could use, other attacks around the lack of a proof that the AI's goals and preferences will remain exactly the same let alone within a window of benevolence over time.
- CuriousSkeptic 11y ago
  How about positive feedback loops? Could we not employ ai-technology as an aid in designing, and implementing, benevolent ai?
  - mercer 11y ago
    That strikes me as sort of circular in reasoning. To create benevolent AI using AI, the latter would already have to know what 'benevolent' means. In which case we might as well just create benevolent AI in the first place. In fact, by relying on a secondary AI it's quite possible we're even more likely to accidentally end up with AI that isn't benevolent.
    - CuriousSkeptic 11y ago
      Ok, bad choice of words. How about augmented reasoning capacity enabled by technologies involved in creating AI.
  - cobaltblue 11y ago
    We could, maybe, for various degrees of help and with various dangers depending on how much help (look up Oracle AI), and I would easily wager using relevant tools (not just "AI tech") would make the positive outcome more likely than not using them, but there are still many ways it can go wrong, and while past and present weak-AI technology might be useful in creating a sentient machine I'm doubtful much would be that helpful in solving the Friendliness (or benevolence) problem directly. For instance the other comment brings up the uncertainty around giving an Oracle AI the task of finding what benevolence means to humans, to then plug into the final AI hoping it will be Friendly. You don't know how to precisely define 'benevolent' so let the Oracle AI do it (or help you do it), though you think you can at least program the Oracle AI to do a good job in finding answers to vague, complex, fragile problems. (Why? Past success in AI tech?) Is what the Oracle AI outputs actually good? How do you know? What did the Oracle AI have to do to reach its answer, or gain a last bit of certainty in it? (e.g. Did it reason it needed more data and so built or hastened the arrival of brain-scanning technology, started scanning brains from volunteers, the very recently deceased, cryonics patients, or just without its creators knowledge through black market deals, and at some point did it simulate trillions of human minds under all sorts of gruesome scenarios to test responses? Do you even morally care about sentience on silicon? Would you care more if you were an upload?) You've got a lot of problems just building an Oracle AI (or anything powerful enough that can help you solve the hardest problems). Even supposing the output it gives is good and the costs (practical and ethical) were acceptable, will a seed AI allowed to recursively self-improve be guaranteed to preserve this Friendliness property in all subsequent improvements? With the idea of using "AI tech", you've basically drawn a line in the outcome space that on one side says something like "let's just program the thing" and the other says "hold on, let's just program the thing but also use all these other relevant computer tools we've developed over the decades to help program the thing". It's the start of an approach, but does it really prune all that much? What do these relevant tools actually buy you in safety, if a safety-guarantee-tool is not among them? Another line which may be what you were heading towards, could be on one side say something like "we might not be able to solve all the necessary problems with our current intelligence, so let's also spend time looking for ways to augment human intelligence -- brain-computer interfaces, brain emulations, tweaks to our genetic code, that sort of thing" and the other side "no, we're capable, let's try our best right now". Using ems based on friendly-ish human minds that become better than human could very well be a better approach to solving the superintelligent problems correctly and efficiently than doing so on raw human levels, on the other hand it could turn very bad if one of those ems instead just solves the problem of recursively improving themselves and their goal and value system is insufficiently friendly. Or many other failure modes. (Though an interesting "most likely scenario" given certain assumptions and only looking at a span of 2 years which by itself doesn't look too terrible is Robin Hanson's Age of Em idea.) Just like approaches involving molecular nanotech to help us, there's reasons why it could be seen to help the best outcome along, but it also opens the door to a lot of other risks that aren't necessarily there if you took the other side, so again how much is really pruned? What I take as your general idea of "maybe we need and should employ additional help and knowledge that we've yet to gain to even really start" isn't bad, though, much better than "Benevolence is easy, a King and his subjects all know he is a great ruler when everyone is smiling frequently", it's just lacking in detail to constitute a plan. Throwing in the positive feedback loop idea (which might even manifest as impressive accelerating returns, who knows) to me seems only relevant to figuring out time scales, I don't think it says anything about the fraction of good vs. bad outcomes apart from whether short or long time scales matter.

Houshalter 11y ago

The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. They would have godlike powers and abilities. And this could happen very rapidly, as the first smart AIs design even better AIs and better computers, which then design even better AIs, and so on. The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. An AI programmed to collect as many paperclips as possible, would convert the mass of the solar system into paperclips. An AI programmed to solve the Riemann hypothesis would build the biggest computer possible to do more calculations. An AI programmed to value self preservation would try to destroy anything that was even a tiny percent chance of being a threat to it. It would try to preserve as much energy and mass as possible to last through the heat death of the universe. Etc. The only AIs which actually do things we want them to do would have to be explicitly programmed to want to do that. And we have no idea how to do that. We can't just hope that arbitrary AIs will happen to value humans instead of something else. We need to figure out how to control them, which requires solving something called "the control problem".

maldusiecle 11y ago
> The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. This is where this line of reasoning completely falls apart for me. Any complex system is going to do unexpected things that weren't deliberately designed for. My laptop computed doesn't do "whatever it was programmed to, to the point of absurdity." Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way?
- TeMPOraL 11y ago
  Nah, that's the key. Your laptop is not doing what it was designed to, it is doing things exactly the way it was built. That's the 101 of programming - programs follow what you wrote, not what you meant. > Why would you expect a program complex enough to simulate the human mind to act in such a predictable way, when none of the much simpler programs we already have act that way? Exactly. We suck at making complex programs do what we want them to. This is the source of worry.
- Houshalter 11y ago
  Your laptop computer does do what it was programmed to do, to the point of absurdity. It will keep executing a computer bug over and ever. Despite the fact that isn't what you intended, it still executes the code anyway. An AI will continue to execute, even when it stops doing what we intended it to do. Computers don't care about our original intentions, they just execute code. A program "complex enough to simulate the human mind" would probably have even more bugs that result in unintentional behavior. Not fewer.
stepvhen 11y ago
>The tl;dr is that we will eventually have AIs which are smarter than humans. Much smarter than humans, to the point that we are to them what chimpanzees are to us. >The second point is that they would have no reason to keep us around. Questions: (1) What reason do we have to keep chimpanzee's around? Why don't we just kill them all? They can't even do calculus. (2) Whatever the answer for question 1, wouldn't an AI consider that same scenario regarding humans? Assuming an AI is truly smarter than a human? (3) If AI gets to the point where they are much smarter than humans, wouldn't it follow that there would be AI philosophers? Or activists? Or pacifists? Would there be such a thing as an AI civilization? Or is your scenario just a few AI overlords that some how slipped through the cracks of regulation by humans?
- TeMPOraL 11y ago
  1. Because we have empathy and because we have no compelling economic reason to (economics always overruling silly things like empathy). 2. Empathy, which is a part of our emotions, is an incredibly complex thing we can't even begin to formalize yet - it's extremely unlikely that a random AI we'd build would happen to have the exact same emotional makeup as we do. 3. "Smart" has a particular meaning here when used in discussing AI. It implies raw intelligence, not morals. A smart AI may be incredibly good at evaluating information, inferring new facts and doing complex tasks. None of it means it will necessarily have to ask itself philosophical questions that we do - those actually mostly come from our emotions, from our desire to understand our place in the universe. The core of the AI-threat issue is that for an AI to not be dangerous it would have to value us and the same things we value. We can't agree about our exact values even among ourselves, and we're nowhere near even trying to formalize it on a level of detail that would be useful for programming.
  - mcguire 11y ago
    That's a good point. How is AI worse than a smart, amoral human?
    - ddingus 11y ago
      It may have very long time frames to work in, may be faster or able to evade the nefarious detection intent of humans by working in ways very different from how we work. Read a book recently that contained an interesting AI attack. Basically, it set a lot of small objects in motion, with complex orbits all computed to result in a large impact at some point in the future. We would not notice something done this way, by analogy.
mcguire 11y ago
I hate to be the logical positivist here (I really, really do), but I'm not sure what your first two sentences mean. What would "smarter than humans" mean and how exactly is it going to happen?
- CuriousSkeptic 11y ago
  My take on how it's going to happen is that it allready has. Most technologies we build, and use, surpass humans in some capacity. Thinking is no exceptions and we allready have machines far better than humans at some thnking tasks, such as arithmetic. We are now working on systems surpassing humans in various recognition and prediction tasks. I suspect the catalog of tasks far better done by machines will continue to expand to finally encompass all tasks required to form a general intelligence. Only, the trick to assemble them, into a complete system could very well prove tricky. The result, though, is that there will never exist such a thing as human level artificial intelligence because, if and when, a system of general intelligence is created all its components will far surpass human equivalent capabilities. So "smarter" simply means far superior to any human in any task requiring thinking, in the same sense that computers are far superior to any human at arithmetic. Even some odd genius savant of today cannot compete with computers at number crunching. More interestingly it's not even interesting to compare, the difference in potential performance is just at different scales all together.

cobaltblue 11y ago

I think a shorter tl;dr is simply that the ways a human-level or beyond AI can be programmed incorrectly (it eventually kills us all or worse), even trying our best to do it right, outnumber the ways it can be programmed correctly. If you don't believe this, I'm not sure how to convince you, other than to poke holes in your ideas about how a benevolent AI could be programmed correctly so you see that such an idea is actually a terrible one or at least insufficient. There are many level of detail attacks I could use, other attacks around the lack of a proof that the AI's goals and preferences will remain exactly the same let alone within a window of benevolence over time.

CuriousSkeptic 11y ago
How about positive feedback loops? Could we not employ ai-technology as an aid in designing, and implementing, benevolent ai?
- mercer 11y ago
  That strikes me as sort of circular in reasoning. To create benevolent AI using AI, the latter would already have to know what 'benevolent' means. In which case we might as well just create benevolent AI in the first place. In fact, by relying on a secondary AI it's quite possible we're even more likely to accidentally end up with AI that isn't benevolent.
  - CuriousSkeptic 11y ago
    Ok, bad choice of words. How about augmented reasoning capacity enabled by technologies involved in creating AI.
- cobaltblue 11y ago
  We could, maybe, for various degrees of help and with various dangers depending on how much help (look up Oracle AI), and I would easily wager using relevant tools (not just "AI tech") would make the positive outcome more likely than not using them, but there are still many ways it can go wrong, and while past and present weak-AI technology might be useful in creating a sentient machine I'm doubtful much would be that helpful in solving the Friendliness (or benevolence) problem directly. For instance the other comment brings up the uncertainty around giving an Oracle AI the task of finding what benevolence means to humans, to then plug into the final AI hoping it will be Friendly. You don't know how to precisely define 'benevolent' so let the Oracle AI do it (or help you do it), though you think you can at least program the Oracle AI to do a good job in finding answers to vague, complex, fragile problems. (Why? Past success in AI tech?) Is what the Oracle AI outputs actually good? How do you know? What did the Oracle AI have to do to reach its answer, or gain a last bit of certainty in it? (e.g. Did it reason it needed more data and so built or hastened the arrival of brain-scanning technology, started scanning brains from volunteers, the very recently deceased, cryonics patients, or just without its creators knowledge through black market deals, and at some point did it simulate trillions of human minds under all sorts of gruesome scenarios to test responses? Do you even morally care about sentience on silicon? Would you care more if you were an upload?) You've got a lot of problems just building an Oracle AI (or anything powerful enough that can help you solve the hardest problems). Even supposing the output it gives is good and the costs (practical and ethical) were acceptable, will a seed AI allowed to recursively self-improve be guaranteed to preserve this Friendliness property in all subsequent improvements? With the idea of using "AI tech", you've basically drawn a line in the outcome space that on one side says something like "let's just program the thing" and the other says "hold on, let's just program the thing but also use all these other relevant computer tools we've developed over the decades to help program the thing". It's the start of an approach, but does it really prune all that much? What do these relevant tools actually buy you in safety, if a safety-guarantee-tool is not among them? Another line which may be what you were heading towards, could be on one side say something like "we might not be able to solve all the necessary problems with our current intelligence, so let's also spend time looking for ways to augment human intelligence -- brain-computer interfaces, brain emulations, tweaks to our genetic code, that sort of thing" and the other side "no, we're capable, let's try our best right now". Using ems based on friendly-ish human minds that become better than human could very well be a better approach to solving the superintelligent problems correctly and efficiently than doing so on raw human levels, on the other hand it could turn very bad if one of those ems instead just solves the problem of recursively improving themselves and their goal and value system is insufficiently friendly. Or many other failure modes. (Though an interesting "most likely scenario" given certain assumptions and only looking at a span of 2 years which by itself doesn't look too terrible is Robin Hanson's Age of Em idea.) Just like approaches involving molecular nanotech to help us, there's reasons why it could be seen to help the best outcome along, but it also opens the door to a lot of other risks that aren't necessarily there if you took the other side, so again how much is really pruned? What I take as your general idea of "maybe we need and should employ additional help and knowledge that we've yet to gain to even really start" isn't bad, though, much better than "Benevolence is easy, a King and his subjects all know he is a great ruler when everyone is smiling frequently", it's just lacking in detail to constitute a plan. Throwing in the positive feedback loop idea (which might even manifest as impressive accelerating returns, who knows) to me seems only relevant to figuring out time scales, I don't think it says anything about the fraction of good vs. bad outcomes apart from whether short or long time scales matter.