4 ms·

In my opinion, all the talks about the potential danger of advanced AI is highly speculative and revolves around a very simple thing: fear of the unknown, that'

In my opinion, all the talks about the potential danger of advanced AI is highly speculative and revolves around a very simple thing: fear of the unknown, that's all. We simply don't know. And some people are also afraid of creation by accident, because intelligence is seen as an emergent property of complex networks, but again, this is because we don't understand much about it. Tldr; Nothing to see here, move along.

qayxc 6y ago
The fears and warnings of the AI safety guys are real and closer than you think. Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results. It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated into the future. Optimisation algorithms are impartial to humans, their moral code and -survival. Simple (and often quoted) example to illustrate this point: say we create an autonomous machine that we want to help us with our stamp collection. We run an optimising algorithm and naively use the amount of unique and rare stamps collected as our target function. The loss function is determined by time taken and money spent. Possible outcomes: the algorithm figures that the most effective way of achieving this is to hack a number of bank accounts to quickly get enough money to buy a nice collection. Now that's bad, but maybe something we can incorporate in our training procedure: only use the money given to you. Another possible outcome: the machine creates a set of robots that roam the planet and steal all the stamps from all over the world. Again, you'd have to consider that in your training method: no stealing. But in the end, you cannot foresee every possible outcome, especially since you expect the machine to come up with an unconventional solution, since otherwise you wouldn't need it in the first place. Restricting the space of possible solutions to safe and desired ones, is a very hard (and potentially undecidable) problem. This paper is just another reminder that we have to be very careful lest we accidentally end human civilisation by means of AI.
- klmadfejno 6y ago
  This is a bad example. The intelligence needed to carry out your described outcomes is more or less AGI already, and that is advanced enough that describing it as a dumb optimization machine is likely naive. Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. It can make trades on stampcollectors.com, for +/- 20% of asking price, max 30 trades per day. Yes, there are some funny examples of RL algos finding weird hacks to beat their simulation, but let's not get ahead of ourselves. We will not see an AI that, from scratch, can learn how to hack bank accounts to buy stamps. The number of nth order observations you need before one can even consider trying to hack a bank account to purchase stamps is ludicrous. If it is even possible, it pales in comparison to the likelihood of someone just making a bot to hack banks to begin with.
  - qayxc 6y ago
    > The intelligence needed to carry out your described outcomes is more or less AGI already Yes! And that's exactly the point at which AI safety comes into play. Nobody is afraid a of a dumb NN that generates texts or images given an input or a classifier. That's not what the paper is concerned with, either. > Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. Again, the example is not concerned with a dumb computation graph. The example is about an AGI with access to the physical world, like robots that need to interact with their environment. The idea of superintelligence even being possible without this interaction (the paper uses "current state of the world as input" to express this) cannot be justified. AI safety is geared toward machines that are capable of manipulating their environment, be that directly (i.e. using physical manipulators) or indirectly (e.g. by controlling external inputs and outputs, communicating with humans and other machines, etc.). If you need a very tangible example that is currently being developed and not at all in realm of sci-fy, look to Japan. The Japanese government recently started an initiative to develop autonomous machines to help caring for their aging population. Such automated caretakers cannot be programmed in a traditional way and are likely to be trained using (unsupervised) reinforcement learning in both virtual and physical environments. Seeing how optimisers love to cheat and game the system in unexpected ways, guaranteeing safety is a real concern in this area and not decades away either. Your average Rumba might be a harmless toy, add an arm and the ability to use an oven to it, and we're talking about a serious accident waiting to happen if we're not careful...
nahuel0x 6y ago
Sometimes fear of the unknown is very useful for survival purposes. There is a reason we evolved with it.

qayxc 6y ago

The fears and warnings of the AI safety guys are real and closer than you think. Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results. It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated into the future. Optimisation algorithms are impartial to humans, their moral code and -survival. Simple (and often quoted) example to illustrate this point: say we create an autonomous machine that we want to help us with our stamp collection. We run an optimising algorithm and naively use the amount of unique and rare stamps collected as our target function. The loss function is determined by time taken and money spent. Possible outcomes: the algorithm figures that the most effective way of achieving this is to hack a number of bank accounts to quickly get enough money to buy a nice collection. Now that's bad, but maybe something we can incorporate in our training procedure: only use the money given to you. Another possible outcome: the machine creates a set of robots that roam the planet and steal all the stamps from all over the world. Again, you'd have to consider that in your training method: no stealing. But in the end, you cannot foresee every possible outcome, especially since you expect the machine to come up with an unconventional solution, since otherwise you wouldn't need it in the first place. Restricting the space of possible solutions to safe and desired ones, is a very hard (and potentially undecidable) problem. This paper is just another reminder that we have to be very careful lest we accidentally end human civilisation by means of AI.

klmadfejno 6y ago
This is a bad example. The intelligence needed to carry out your described outcomes is more or less AGI already, and that is advanced enough that describing it as a dumb optimization machine is likely naive. Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. It can make trades on stampcollectors.com, for +/- 20% of asking price, max 30 trades per day. Yes, there are some funny examples of RL algos finding weird hacks to beat their simulation, but let's not get ahead of ourselves. We will not see an AI that, from scratch, can learn how to hack bank accounts to buy stamps. The number of nth order observations you need before one can even consider trying to hack a bank account to purchase stamps is ludicrous. If it is even possible, it pales in comparison to the likelihood of someone just making a bot to hack banks to begin with.
- qayxc 6y ago
  > The intelligence needed to carry out your described outcomes is more or less AGI already Yes! And that's exactly the point at which AI safety comes into play. Nobody is afraid a of a dumb NN that generates texts or images given an input or a classifier. That's not what the paper is concerned with, either. > Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. Again, the example is not concerned with a dumb computation graph. The example is about an AGI with access to the physical world, like robots that need to interact with their environment. The idea of superintelligence even being possible without this interaction (the paper uses "current state of the world as input" to express this) cannot be justified. AI safety is geared toward machines that are capable of manipulating their environment, be that directly (i.e. using physical manipulators) or indirectly (e.g. by controlling external inputs and outputs, communicating with humans and other machines, etc.). If you need a very tangible example that is currently being developed and not at all in realm of sci-fy, look to Japan. The Japanese government recently started an initiative to develop autonomous machines to help caring for their aging population. Such automated caretakers cannot be programmed in a traditional way and are likely to be trained using (unsupervised) reinforcement learning in both virtual and physical environments. Seeing how optimisers love to cheat and game the system in unexpected ways, guaranteeing safety is a real concern in this area and not decades away either. Your average Rumba might be a harmless toy, add an arm and the ability to use an oven to it, and we're talking about a serious accident waiting to happen if we're not careful...

nahuel0x 6y ago

Sometimes fear of the unknown is very useful for survival purposes. There is a reason we evolved with it.