3 ms·
The fears and warnings of the AI safety guys are real and closer than you think. Machine superintelligence is a bit on the extreme side, but formulating safety
by qayxc 6y ago
The fears and warnings of the AI safety guys are real and closer than you think.
Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results.
It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated into the future. Optimisation algorithms are impartial to humans, their moral code and -survival.
Simple (and often quoted) example to illustrate this point: say we create an autonomous machine that we want to help us with our stamp collection. We run an optimising algorithm and naively use the amount of unique and rare stamps collected as our target function. The loss function is determined by time taken and money spent.
Possible outcomes: the algorithm figures that the most effective way of achieving this is to hack a number of bank accounts to quickly get enough money to buy a nice collection.
Now that's bad, but maybe something we can incorporate in our training procedure: only use the money given to you.
Another possible outcome: the machine creates a set of robots that roam the planet and steal all the stamps from all over the world.
Again, you'd have to consider that in your training method: no stealing.
But in the end, you cannot foresee every possible outcome, especially since you expect the machine to come up with an unconventional solution, since otherwise you wouldn't need it in the first place.
Restricting the space of possible solutions to safe and desired ones, is a very hard (and potentially undecidable) problem. This paper is just another reminder that we have to be very careful lest we accidentally end human civilisation by means of AI.
- klmadfejno 6y agoThis is a bad example. The intelligence needed to carry out your described outcomes is more or less AGI already, and that is advanced enough that describing it as a dumb optimization machine is likely naive. Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. It can make trades on stampcollectors.com, for +/- 20% of asking price, max 30 trades per day. Yes, there are some funny examples of RL algos finding weird hacks to beat their simulation, but let's not get ahead of ourselves. We will not see an AI that, from scratch, can learn how to hack bank accounts to buy stamps. The number of nth order observations you need before one can even consider trying to hack a bank account to purchase stamps is ludicrous. If it is even possible, it pales in comparison to the likelihood of someone just making a bot to hack banks to begin with.
- qayxc 6y ago> The intelligence needed to carry out your described outcomes is more or less AGI already Yes! And that's exactly the point at which AI safety comes into play. Nobody is afraid a of a dumb NN that generates texts or images given an input or a classifier. That's not what the paper is concerned with, either. > Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. Again, the example is not concerned with a dumb computation graph. The example is about an AGI with access to the physical world, like robots that need to interact with their environment. The idea of superintelligence even being possible without this interaction (the paper uses "current state of the world as input" to express this) cannot be justified. AI safety is geared toward machines that are capable of manipulating their environment, be that directly (i.e. using physical manipulators) or indirectly (e.g. by controlling external inputs and outputs, communicating with humans and other machines, etc.). If you need a very tangible example that is currently being developed and not at all in realm of sci-fy, look to Japan. The Japanese government recently started an initiative to develop autonomous machines to help caring for their aging population. Such automated caretakers cannot be programmed in a traditional way and are likely to be trained using (unsupervised) reinforcement learning in both virtual and physical environments. Seeing how optimisers love to cheat and game the system in unexpected ways, guaranteeing safety is a real concern in this area and not decades away either. Your average Rumba might be a harmless toy, add an arm and the ability to use an oven to it, and we're talking about a serious accident waiting to happen if we're not careful...
- klmadfejno 6y agoYou're making the assumption that AGI level intelligence would resemble RL bots of today. It's a classic sci fi trope, but likely not one that makes any sense imo. AGI might require physical instantiation, but that doesn't mean that physical bots are indicative of AGI outcomes. Most of our physical bots are just classically written programs with a bit of ML to help with classification of its environment anyway.