4 ms·
This is a really important problem to solve, for more than just self driving cars. We have no good way of controlling AIs right now. Particularly reinforcement
by Houshalter 9y ago
This is a really important problem to solve, for more than just self driving cars. We have no good way of controlling AIs right now. Particularly reinforcement learners. All we can do is make a fitness function and hope they maximize it. Often they will find a way to exploit it in a way the programmer didn't expect or desire.
And even making a fitness function is a pretty difficult task. You can't exactly let a self driving car get into thousands of accidents until it learns not to do that. Making a good simulation is extremely difficult and very expensive. Training it to mimic humans means it will always be worse than humans.
As AIs get smarter this will become more of a problem. They will become more clever at exploiting the reward functions and even fooling human supervisors. One could imagine a self driving car causing accidents indirectly in some way. But it doesn't care because the car itself didn't collide with anything.
In the extreme case, AIs much smarter than humans would be extremely difficult to trust. They could figure out how to manipulate humans to get them to press their reward buttons or whatever equivalent. The famous thought experiment is an AI in a paperclip factory that desires to maximize paperclip production. So it invents nanotech, tears apart the Earth, and eventually turns the entire solar system into a massive paperclip factory.
Perhaps that's a long way off. But it's not comforting to know that AI ability is increasing faster than our ability to control AI.
- cbames89 9y ago"Training it to mimic humans means it will always be worse than humans." This is not necessarily true, there is work in learning from demonstration where robots have exceed the "experts" reward. I'll post some papers later if you're interested.
- YeGoblynQueenne 9y agoPlease do. I'm not the OP but I'm interested.
- cbames89 9y agoSorry for the delay. Here's some early work from Pieter Abbeel where a helicopter does aerobatic manuevers better than the demonstrations it received: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=EAEFFAD48896E51495A1F4192E6A4F74?doi=10.1.1.140.3114&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=EAE...
- chongli 9y agoTraining it to mimic humans means it will always be worse than humans. The success of AlphaGo says otherwise. The distinction is that you're not mimicking one human but some large set of humans and you're using your fitness function to guide you toward a maximum 'meta human'. We don't need self driving cars to be perfect drivers. If they're better than any human driver then that's a good enough reason to replace human drivers.
- deleted 9y ago[deleted]
- akvadrako 9y agoAlphaGo was not primarily trained to mimic humans, but to win games. This included playing many games against itself and semi-random tree searches for better strategies. If it was only mimicking humans it probably would have lost to the world's best.
- Houshalter 9y agoAlphaGo was only initialized by mimicking humans. It then played millions of games against itself. And learned what moves were more likely to lead to wins. And then it was combined with a tree search algorithm that let it explore many more moves into the future than humans could consider. Training a bot just to predict what move a human would make, would always lose to that human. It will just absorb all the human's mistakes and weaknesses. And on top of that it will add it's own, since no AI algorithm is anywhere near 100% perfect.