4 ms·

Probability of AGI is close to 100%, conditional on mankind surviving long enough. We know that human brains don't run on magic, therefore you can build a machi

Probability of AGI is close to 100%, conditional on mankind surviving long enough. We know that human brains don't run on magic, therefore you can build a machine that can do the same thing. (Turing computability doesn't matter, if the brain relies on non-Turing-computable physical effects then a machine can rely on those as well.) The current situation where unmodified humans are the smartest creatures around is unstable in the long term. Saying that we don't have to worry because AI development will take 1000 years is also missing the point. Research into unsafe AI is inherently faster than research into safe AI, because the latter problem is more constrained. If unsafe AI kills us all, the best time to start working on safe AI is 100 years ago, and the second best time is now. Saying that we don't have to worry because smarter than human intelligences will have a better understanding of morality is also missing the point. All currently known mathematical formalisms for decision-making allow you to plug in arbitrary utility functions, including "immoral" ones. Thinking that future decision-making formalisms will support only "moral" utility functions is extreme wishful thinking. Saying that we don't have to worry because brain emulations or human cognitive enhancement will come first is also missing the point. These things can be just as dangerous as AI (e.g. see Robin Hanson's descriptions of how mind uploading increases Darwinian pressures on everyone, including non-uploaded people), and can also lead to AI more quickly. And so on. If you want to feel safe and protected, no one can stop you. But you don't have any good arguments for feeling safe. I'm sorry, but you really don't. I've been on the lookout for such arguments for years, and I'd be happier if they existed.

AndrewKemendo 12y ago
We know that human brains don't run on magic, therefore you can build a machine that can do the same thing. I mean that is what I think, and most of the community thinks, but we haven't proven that empirically - nor could we, its unprovable. But you don't have any good arguments for feeling safe. Correct. Notice that I am not saying that we are safe. I personally think we aren't safe [1], but that is a different discussion which is equally as baseless. The key here is that the argument that AGI will not be safe, and therefore needs to be regulated to make it safe, not only misses the point but ensures that we would never make one. The current level of funding for AGI development and understanding of the roadmap for AGI is absolutely dismal given it's potential impact. If we now add the constraint that in order to make one, we also ensure it doesn't threaten humanity - then you have sounded the death knell for AGI. Once we make AGI - in the sense that it is a recursively self improving system with human equivalent capabilities across domains - there is no way we will know what will happen. That's it. [1] Which is why I am personally in the transhumanist camp.
- cousin_it 12y ago
  Whether AGI is "tragically underfunded" or "luckily underfunded" is a matter of perspective. If the DoD announced a Manhattan-scale project to create an AI for defense, I'd be more worried than I am now. At the current levels of funding, smaller groups have at least some chance of making progress on safety research. It's also encouraging that Google is taking the problem somewhat seriously, and has instituted an "AI ethics" committee as part of the DeepMind acquisition.
  - AndrewKemendo 12y ago
    Definitely agreed, except for the DoD Manhattan project part - I would love to see that.
eli_gottlieb 12y ago
>Saying that we don't have to worry because smarter than human intelligences will have a better understanding of morality is also missing the point. All currently known mathematical formalisms for decision-making allow you to plug in arbitrary utility functions, including "immoral" ones. Thinking that future decision-making formalisms will support only "moral" utility functions is extreme wishful thinking. I'm going to play Devil's Advocate on this. While it is true that all known decision-making formulas allow for immoral utility functions, I think it's a flaw of current approaches, such as reinforcement learning, that they only allow nonspecific utility functions. I can't write down an AI program to maximize paperclips -- there's no way, in current formalisms, to specify what that means. I think the FAI ideologues and the mainstream machine-learning community can both rally around the prospect of being able to specify a goal in terms other than reinforcement. Scientifically, that's an entirely sensible research question for the field of AI/ML to tackle, and it also constitutes a major step away from the scenarios in which a dangerous AGI winds up accidentally programmed with random, arbitrary goals that nobody actually wanted. After all, fundamental philosophical questions aside, a machine that flies out of control at random when turned on is simply a bad machine.
- cousin_it 12y ago
  Yeah, I agree that reinforcement learning is probably a bad approach to FAI. Most of our toy models involve utility functions encoded directly into the AI, not reinforcement. That said, it's indeed very hard to directly specify a utility function involving paperclips. If our universe were a Game of Life universe and we knew exactly which configuration corresponds to a paperclip, I'd be able to do that right now. But since we don't know the true laws of physics, the "hard way" involves encoding some kind of Solomonoff prior over all possible physical universes, and a rule for recognizing paperclips in each of them. That's kind of a tall order. There's a shortcut that involves encoding a precise mathematical description of a human mind into an AI, and passing the buck by saying "the AI must maximize whatever function the human mind would output, given enough time to think". That would actually be straightforward to implement, if we had a description of a human mind that we were willing to trust. Unfortunately, the naive form of that approach immediately fails due to acausal blackmail. There's no obvious fix, but some folks are trying to devise non-obvious fixes, and I suppose it can be made to work in the long run. Of course, after you can formulate a utility function in terms of a precise description of a human mind, the next step is reformulating it in terms of the outputs of an actually existing person, say some webcam videos and written instructions. You instruct the AI to infer the simplest program that would generate these outputs (using something like the Solomonoff prior again), and then proceed as in the previous step. That part has its own problems, but I expect that it can also be made to work. The whole approach is kind of a long shot, but I hope that I've given a sense that the problem could be solvable by careful human effort on the scale of years. There are other approaches as well.

AndrewKemendo 12y ago

We know that human brains don't run on magic, therefore you can build a machine that can do the same thing. I mean that is what I think, and most of the community thinks, but we haven't proven that empirically - nor could we, its unprovable. But you don't have any good arguments for feeling safe. Correct. Notice that I am not saying that we are safe. I personally think we aren't safe [1], but that is a different discussion which is equally as baseless. The key here is that the argument that AGI will not be safe, and therefore needs to be regulated to make it safe, not only misses the point but ensures that we would never make one. The current level of funding for AGI development and understanding of the roadmap for AGI is absolutely dismal given it's potential impact. If we now add the constraint that in order to make one, we also ensure it doesn't threaten humanity - then you have sounded the death knell for AGI. Once we make AGI - in the sense that it is a recursively self improving system with human equivalent capabilities across domains - there is no way we will know what will happen. That's it. [1] Which is why I am personally in the transhumanist camp.

cousin_it 12y ago
Whether AGI is "tragically underfunded" or "luckily underfunded" is a matter of perspective. If the DoD announced a Manhattan-scale project to create an AI for defense, I'd be more worried than I am now. At the current levels of funding, smaller groups have at least some chance of making progress on safety research. It's also encouraging that Google is taking the problem somewhat seriously, and has instituted an "AI ethics" committee as part of the DeepMind acquisition.
- AndrewKemendo 12y ago
  Definitely agreed, except for the DoD Manhattan project part - I would love to see that.

eli_gottlieb 12y ago

>Saying that we don't have to worry because smarter than human intelligences will have a better understanding of morality is also missing the point. All currently known mathematical formalisms for decision-making allow you to plug in arbitrary utility functions, including "immoral" ones. Thinking that future decision-making formalisms will support only "moral" utility functions is extreme wishful thinking. I'm going to play Devil's Advocate on this. While it is true that all known decision-making formulas allow for immoral utility functions, I think it's a flaw of current approaches, such as reinforcement learning, that they only allow nonspecific utility functions. I can't write down an AI program to maximize paperclips -- there's no way, in current formalisms, to specify what that means. I think the FAI ideologues and the mainstream machine-learning community can both rally around the prospect of being able to specify a goal in terms other than reinforcement. Scientifically, that's an entirely sensible research question for the field of AI/ML to tackle, and it also constitutes a major step away from the scenarios in which a dangerous AGI winds up accidentally programmed with random, arbitrary goals that nobody actually wanted. After all, fundamental philosophical questions aside, a machine that flies out of control at random when turned on is simply a bad machine.

cousin_it 12y ago
Yeah, I agree that reinforcement learning is probably a bad approach to FAI. Most of our toy models involve utility functions encoded directly into the AI, not reinforcement. That said, it's indeed very hard to directly specify a utility function involving paperclips. If our universe were a Game of Life universe and we knew exactly which configuration corresponds to a paperclip, I'd be able to do that right now. But since we don't know the true laws of physics, the "hard way" involves encoding some kind of Solomonoff prior over all possible physical universes, and a rule for recognizing paperclips in each of them. That's kind of a tall order. There's a shortcut that involves encoding a precise mathematical description of a human mind into an AI, and passing the buck by saying "the AI must maximize whatever function the human mind would output, given enough time to think". That would actually be straightforward to implement, if we had a description of a human mind that we were willing to trust. Unfortunately, the naive form of that approach immediately fails due to acausal blackmail. There's no obvious fix, but some folks are trying to devise non-obvious fixes, and I suppose it can be made to work in the long run. Of course, after you can formulate a utility function in terms of a precise description of a human mind, the next step is reformulating it in terms of the outputs of an actually existing person, say some webcam videos and written instructions. You instruct the AI to infer the simplest program that would generate these outputs (using something like the Solomonoff prior again), and then proceed as in the previous step. That part has its own problems, but I expect that it can also be made to work. The whole approach is kind of a long shot, but I hope that I've given a sense that the problem could be solvable by careful human effort on the scale of years. There are other approaches as well.