7 ms·

AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today'

AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today's billionaires. Couple this with extended life spans and intelligence enhancement, and the entire (human) power structure as we know it today is going to melt away like an iceberg into the vast oceans. Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know. But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way. I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"? And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production. >> nok (is-good-idea (turn-humans-into-paperclips)) Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.

jessriedel 12y ago
You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced. (Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about. https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192
- jsnathan 12y ago
  Don't take this the wrong way, but this book is precisely the kind of thing I was talking about. That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up. If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious. Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that. [1]: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - jessriedel 12y ago
    The question of whether AI is dangerous in the hands of people with good intentions is more difficult than the one of whether it's dangerous generally. I was only trying to convince you of the former, in response to this: > AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want. The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right. Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is. EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.
PeterisP 12y ago
A self-modifying, self-improving system (the only realistic way how we'll get to a powerful AI) will effectively design, implement and test the final version by itself. It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented. Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.
- jsnathan 12y ago
  > if and only if it wanted be friendly and respect those constraints in the first place No, not if it "wanted" to - if it was written that way. > otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes. But, it would. > Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem. No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I. If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted. Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".
  - PeterisP 12y ago
    This is not something that I just throw out there - these are real research problems with no known good solutions (but a lot of published attempts and discussions) despite having a lot of phd-years thrown at it already. For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc. This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ https://intelligence.org/research/ may be a place to start, if you're intereseted.
AndrewKemendo 12y ago
Interestingly enough I have yet to read a really well thought through scenario where an AGI is devastatingly harmful and I have read pretty much everything there is to read on the subject from Bostrom, Yudkowski, MIRI etc... It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.

jessriedel 12y ago

You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced. (Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about. https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192

jsnathan 12y ago
Don't take this the wrong way, but this book is precisely the kind of thing I was talking about. That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up. If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious. Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that. [1]: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
- jessriedel 12y ago
  The question of whether AI is dangerous in the hands of people with good intentions is more difficult than the one of whether it's dangerous generally. I was only trying to convince you of the former, in response to this: > AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want. The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right. Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is. EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.

PeterisP 12y ago

A self-modifying, self-improving system (the only realistic way how we'll get to a powerful AI) will effectively design, implement and test the final version by itself. It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented. Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.

jsnathan 12y ago
> if and only if it wanted be friendly and respect those constraints in the first place No, not if it "wanted" to - if it was written that way. > otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes. But, it would. > Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem. No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I. If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted. Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".
- PeterisP 12y ago
  This is not something that I just throw out there - these are real research problems with no known good solutions (but a lot of published attempts and discussions) despite having a lot of phd-years thrown at it already. For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc. This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ https://intelligence.org/research/ may be a place to start, if you're intereseted.

AndrewKemendo 12y ago

Interestingly enough I have yet to read a really well thought through scenario where an AGI is devastatingly harmful and I have read pretty much everything there is to read on the subject from Bostrom, Yudkowski, MIRI etc... It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.