3 ms·

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read wa

by nzp 12y ago

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Because they have biological constraints, and a hardwired self preservation drive. How to get that in a machine made by us rather than a billion years of natural process, that's the hard question. How to make a machine with the equivalent of motives and emotions. Both sides of the problem are well recognized. In an ideal world I would prefer this research (AGI, and there is a quantum leap to be made between self driving cars and AGI) to stay strictly under the auspices of “pure science”, i.e. academia. No government, no private interests. But that's not going to happen. The next best thing, I think, would be to not let those in power monopolize the research.

davmre 12y ago
> Not so much what n and k to set, but how to get such an entity to do anything at all. I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act. What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks. Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.
- nzp 12y ago
  You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.

by nzp 12y ago

davmre 12y ago

> Not so much what n and k to set, but how to get such an entity to do anything at all. I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act. What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks. Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.

nzp 12y ago
You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.