3 ms·
That's definitely true, and a major problem beyond friendliness (i.e. way before any singularity). A traditional assumption in machine learning and other areas
by _delirium 14y ago
That's definitely true, and a major problem beyond friendliness (i.e. way before any singularity). A traditional assumption in machine learning and other areas of AI is that you can factor out the problem-specification parts from the algorithmic problem-solving parts, but it's very easy to get results that are unexpected, except in the rare cases where you have an 100%-correct obvious objective function handed to you by the structure of a problem. In cases where a human is writing down what they think the objective is, a very common case is that there are a lot of unspoken things that they intended to be included, "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do". Hence a lot of iteration on objective functions is needed in real-world applications, and recent-ish research focuses on alternative formulations like interactive machine learning, preference elicitation, etc.
- adrianN 14y agoFor an example of this have a look at this essay [1] by George Danzig in which he tells the story of how he tried optimizing his diet using linear programming. [1] http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf http://dl.dropbox.com/u/5317066/1990-dantzig-dietproblem.pdf
- goostavos 14y agoInteresting read. Thanks a lot!
- endtime 14y ago> "optimize [x], but without doing anything obviously stupid that I wouldn't want you to do" This is similar to the basic idea of Yudkowsky et al's Coherent Extrapolated Volition, the gist of which is that an FAI's metaethical function should be some aggregated of what it believes people would want it to do if they had the knowledge and intelligence(/cycles) that the AI has. (I believe there exist updates/improvements to CEV, but I don't know much about them.)