This is not about the size of the system. You really cannot predict how the system will work, since its observing input from the outside and constructing its ac

This is not about the size of the system. You really cannot predict how the system will work, since its observing input from the outside and constructing its actions based on it. Sure, there are ways to ask some ML systems why they made a decision after the fact, and they can elaborate which variables had the most effect. But before the algorithm gets the training data, you don't know what it will decide - nobody does! It picks actions that maximise a value it was told to maximise. It does this based on "well, when I encountered a situation like this other times, I did these 5 things, and this one thing maximised that value I'm told to maximise in most of those cases, so I'll do that. At the beginning it will try random things. After several iterations, it will find things that "stick" i.e. maximise the value. There are sophisticated techniques to get unstuck from local maximums and minimums, too, in order to find even bigger maximums. When you have enough data, enough users, enough actions in a system, you can very quickly try a lot of things and learn a lot of stuff. But what those stuff would be, no human can predict. The main problem is that its unethical to train a machine learning system with a value function that doesn't take into account basic ethical behaviour. There are 3 challenges here The first challenge is technical, and its the easiest one - how to come up with a value function that encodes additional measures of human ethics. Its easy as in, you take a bunch of ethicists, give them various situations and ask them to rate actions as ethical/unethical. Then once you have that, you try to put it into the value function, so that the system can learn some decency. All done. (If only it were that easy!) The second challenge is a business one, which is, how far are you willing to reduce your value maximisation to be ethical, and what to do if your competitor doesn't do that. One way to solve that is to have regulations for ethical behaviour of machine learning systems. The system could be held responsible for unethical actions. If such actions are reported by people, investigated by experts and found true in court, the company is held liable. Public pressure and exposure seems to help too. Perhaps we could make machine learning systems that detect unethical behaviour and call it the ML police. Citizens could agree to install the ML police add-on to help monitor and aggregate behaviour of online ML systems. (If these suggestions look silly, its because they are) The third challenge is philosophical. Until now, philosophers were content with "there is no right answer, but there have been many thoughts on what exactly is ethical". They better get their act together, because we'll need them to come up with a definite, quantifiable answer real soon. On the more optimistic side, lets hope that any generally agreed upon ethical system is better than no system at all.