Imagine making a linear regression to predict runners’ speed using data from two individuals timing their speeds. Not a very interesting problem. Assuming they’

Imagine making a linear regression to predict runners’ speed using data from two individuals timing their speeds. Not a very interesting problem. Assuming they’re both competent measurers, you would probably be best off taking half of the first measure and half of the second measure (aka the mean). But the clever linear solver might notice that you can technically get a better answer by weighting the first measurer by 3.001 and the second measurer by -2.002 due to natural variance and the way things happened to land. We can adjust the solver to not do this by punishing it for large coefficients (ridge regression) and that’ll mostly settle that. But for a neural network with hundreds of mini logistic regressions built in... seems tough. Interpretability isn’t just knowing how decisions are made but also how much irrationality is being built into it. If one factor is being marked as a huge negative influence, what does that actually mean about the problem space? Maybe nothing. If you actually want an interpretable neural net model, you should probably hand craft layers of domain specific ensembles that you can individually verify and are closer to being self evident. Maybe you won’t know how it determines stripes or horselike, but if you feel good about those two models individually then it’s a much easier task to follow the last step which is: it’s a zebra if and only if it’s horselike and has stripes.