4 ms·
1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these
by bootsmann 4y ago
1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
- mitthrowaway2 4y agoOne of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function. (An incentive is just the gradient of the loss function, btw).
- bootsmann 4y agoIt is, the model will behave in a way to minimize (or at least approximate, most aren't convex) the loss function. This isn't magical or unexpected.
- mitthrowaway2 4y agoYes, but when there are many degrees of freedom, that minimization might not happen in the way you expected, and if your loss function was only an approximation to your utility (because you weren't sure how to precisely codify "goodness"), the difference between the behavior you hoped for and the behavior you actually get might be very bad. For example, you attempt train an AI to get good behavior via a penalty button and reward button; an approximately-optimal strategy might include shooting the researcher to prevent him from pressing the penalty button, and then mashing the reward button itself.
- kmeisthax 4y agoI recommend actually watching some videos on the topic before insisting that the problem does not exist: https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI