3 ms·

I recently watched: http://www.youtube.com/watch?v=Uoda5BSj_6o http://www.youtube.com/watch?v=Uoda5BSj_6o in which Eliezer Yudkowsky talks about challenges in

by feral 14y ago

I recently watched: http://www.youtube.com/watch?v=Uoda5BSj_6o http://www.youtube.com/watch?v=Uoda5BSj_6o in which Eliezer Yudkowsky talks about challenges in making 'friendly' AI. Yudkowsky draws a lot of examples from evolution to show how a system blindly optimising for an objective function can result in all sorts of things that a human specifying the objective function wouldn't ever have expected. The linked article at one point says: "In order to be a perfect and beautiful computing machine, it is not requisite to know what arithmetic is." After watching Yudkowsky's talk, which I found very insightful, I got the impression that it could be very difficult to make a machine that wouldn't do very unpredictable things, unless you had a very deep understanding of exactly what you were telling it to do. The talk might be of interest.

_delirium 14y ago
That's definitely true, and a major problem beyond friendliness (i.e. way before any singularity). A traditional assumption in machine learning and other areas of AI is that you can factor out the problem-specification parts from the algorithmic problem-solving parts, but it's very easy to get results that are unexpected, except in the rare cases where you have an 100%-correct obvious objective function handed to you by the structure of a problem. In cases where a human is writing down what they think the objective is, a very common case is that there are a lot of unspoken things that they intended to be included, "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do". Hence a lot of iteration on objective functions is needed in real-world applications, and recent-ish research focuses on alternative formulations like interactive machine learning, preference elicitation, etc.
- adrianN 14y ago
  For an example of this have a look at this essay [1] by George Danzig in which he tells the story of how he tried optimizing his diet using linear programming. [1] http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf
  - goostavos 14y ago
    Interesting read. Thanks a lot!
- endtime 14y ago
  > "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do" This is similar to the basic idea of Yudkowsky et al's Coherent Extrapolated Volition, the gist of which is that an FAI's metaethical function should be some aggregated of what it believes people would want it to do if they had the knowledge and intelligence(/cycles) that the AI has. (I believe there exist updates/improvements to CEV, but I don't know much about them.)
Jun8 14y ago
"At one point in the discussion, Sussman told Minsky that he was using a certain randomizing technique in his program because he didn't want the machine to have any preconceived notions. Minsky said, "Well, it has them, it's just that you don't know what they are." It was the most profound thing Gerry Sussman had ever heard. "

by feral 14y ago

_delirium 14y ago

That's definitely true, and a major problem beyond friendliness (i.e. way before any singularity). A traditional assumption in machine learning and other areas of AI is that you can factor out the problem-specification parts from the algorithmic problem-solving parts, but it's very easy to get results that are unexpected, except in the rare cases where you have an 100%-correct obvious objective function handed to you by the structure of a problem. In cases where a human is writing down what they think the objective is, a very common case is that there are a lot of unspoken things that they intended to be included, "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do". Hence a lot of iteration on objective functions is needed in real-world applications, and recent-ish research focuses on alternative formulations like interactive machine learning, preference elicitation, etc.

adrianN 14y ago
For an example of this have a look at this essay [1] by George Danzig in which he tells the story of how he tried optimizing his diet using linear programming. [1] http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf
- goostavos 14y ago
  Interesting read. Thanks a lot!
endtime 14y ago
> "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do" This is similar to the basic idea of Yudkowsky et al's Coherent Extrapolated Volition, the gist of which is that an FAI's metaethical function should be some aggregated of what it believes people would want it to do if they had the knowledge and intelligence(/cycles) that the AI has. (I believe there exist updates/improvements to CEV, but I don't know much about them.)

Jun8 14y ago

"At one point in the discussion, Sussman told Minsky that he was using a certain randomizing technique in his program because he didn't want the machine to have any preconceived notions. Minsky said, "Well, it has them, it's just that you don't know what they are." It was the most profound thing Gerry Sussman had ever heard. "