In many cases the programmer could not have made the decisions. For example, AlphaGo programmers could not win against top go professionals, but AlphaGo can. W

In many cases the programmer could not have made the decisions. For example, AlphaGo programmers could not win against top go professionals, but AlphaGo can. With a simple model like credit score it's easy to see which input affects which output in which direction. You're late on your mortgage - your FICO score drops. But this is just one neuron. Make the model a little more complex, and you no longer have a way to know what exactly it is doing.