4 ms·
The Base Assumptions undergirding all AI safety research is that: 1. AI systems respond to incentives, not commands, including incentives you did not intend to
by kmeisthax 4y ago
The Base Assumptions undergirding all AI safety research is that:
1. AI systems respond to incentives, not commands, including incentives you did not intend to be there.
2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible.
#1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0].
#2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc).
My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed.
On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us.
[0] So are human children, but they do not have the I/O capability of a data center full of A100s.
[1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.
- bootsmann 4y ago1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
- mitthrowaway2 4y agoOne of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function. (An incentive is just the gradient of the loss function, btw).
- bootsmann 4y agoIt is, the model will behave in a way to minimize (or at least approximate, most aren't convex) the loss function. This isn't magical or unexpected.
- mitthrowaway2 4y agoYes, but when there are many degrees of freedom, that minimization might not happen in the way you expected, and if your loss function was only an approximation to your utility (because you weren't sure how to precisely codify "goodness"), the difference between the behavior you hoped for and the behavior you actually get might be very bad. For example, you attempt train an AI to get good behavior via a penalty button and reward button; an approximately-optimal strategy might include shooting the researcher to prevent him from pressing the penalty button, and then mashing the reward button itself.
- kmeisthax 4y agoI recommend actually watching some videos on the topic before insisting that the problem does not exist: https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI
- 13years 4y ago> and according to #1, we will always get it wrong This is a point I've made. We are playing with paradoxes that proponents tend to ignore. They won't be solvable. The more sophisticated AI becomes, the more non-deterministic behavior arises. The emergent deception has been very interesting. https://bounded-regret.ghost.io/emergent-deception-optimization https://bounded-regret.ghost.io/emergent-deception-optimizat... I've elaborated on #2 much further here as well - https://dakara.substack.com/p/ai-singularity-the-hubris-trap https://dakara.substack.com/p/ai-singularity-the-hubris-trap