4 ms·

What you quote here is in no way in disagreement with what fchollet quoted. I strictly agree with this. No one is denying the need to not create a raving supe

by nzp 12y ago

What you quote here is in no way in disagreement with what fchollet quoted. I strictly agree with this. No one is denying the need to not create a raving super-intelligent psychopath. The problem with this fear mongering is that it's not based on any fact, namely, implying that AGI (or SMI, whatever) would by default (naturally even!) be hostile or negligently indifferent to biological life. It could very well be that it would naturally tend to be very friendly, or at least benevolently indifferent. Why would such an entity try to destroy humans? Not only that, the proposal is to get governments, the US government especially—an organization which in total has not been waging war for only a few decades during it's entire existence—in explicit and tight control of this powerful technology. Doesn't quite seem to be the best idea around, frankly.

davmre 12y ago
There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values...". Nick Bostrom and others build this out in much greater detail. There are lots of reasons to be skeptical of government regulation. It would probably be nicer if industry can self-regulate and allow governments to stick to the positive role of funding research into the technical questions of building provably controllable/friendly decision-theoretic systems. But a lot of people are claiming that the unified consensus of AI experts is "there's no problem here at all", and I think it's important to push back against that.
- nzp 12y ago
  Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Because they have biological constraints, and a hardwired self preservation drive. How to get that in a machine made by us rather than a billion years of natural process, that's the hard question. How to make a machine with the equivalent of motives and emotions. Both sides of the problem are well recognized. In an ideal world I would prefer this research (AGI, and there is a quantum leap to be made between self driving cars and AGI) to stay strictly under the auspices of “pure science”, i.e. academia. No government, no private interests. But that's not going to happen. The next best thing, I think, would be to not let those in power monopolize the research.
  - davmre 12y ago
    > Not so much what n and k to set, but how to get such an entity to do anything at all. I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act. What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks. Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.
    - nzp 12y ago
      You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.

by nzp 12y ago

davmre 12y ago

There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values...". Nick Bostrom and others build this out in much greater detail. There are lots of reasons to be skeptical of government regulation. It would probably be nicer if industry can self-regulate and allow governments to stick to the positive role of funding research into the technical questions of building provably controllable/friendly decision-theoretic systems. But a lot of people are claiming that the unified consensus of AI experts is "there's no problem here at all", and I think it's important to push back against that.

nzp 12y ago
Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Because they have biological constraints, and a hardwired self preservation drive. How to get that in a machine made by us rather than a billion years of natural process, that's the hard question. How to make a machine with the equivalent of motives and emotions. Both sides of the problem are well recognized. In an ideal world I would prefer this research (AGI, and there is a quantum leap to be made between self driving cars and AGI) to stay strictly under the auspices of “pure science”, i.e. academia. No government, no private interests. But that's not going to happen. The next best thing, I think, would be to not let those in power monopolize the research.
- davmre 12y ago
  > Not so much what n and k to set, but how to get such an entity to do anything at all. I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act. What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks. Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.
  - nzp 12y ago
    You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.