I convinced that neural network will never be interpretable like code. They will just get even more complex, so to employ them in critical applications (my moti

I convinced that neural network will never be interpretable like code. They will just get even more complex, so to employ them in critical applications (my motivation is more healthcare, where similiar constraints are present. People can die if there's a wrong prediction) we have to build a notion why we allow something and what we don't allow without some human in the loop getting convined that the reasoning itself is sound. I think this boils down to assumptions about robustness (like the l2-norm adversarial examples in image classification) and statistics. to 1). Neural networks are code, they are always 100% the same if you ship it. So you can test them way more thoughly than some equipment, which (i hope) would allow you to minimze the chance of failure to a reasonable degree. to 2) "you don't know when or why it won't work." I think a fundamental disagreement lies here. I am convinced that this is not possible with these complex function approximators, where we just optimize them until we are happy. If we adopt a thinking like above, we can retrain the networks with these additional inputs, adjust our notion of robustness and think whether there's a flaw in our statistical assumptions. But these are just random thoughts. I am in touch with people working in healthcare with this stuff, so there's some exposure to these kinds of problems. But I have never read any work discussing these issues or really reflection whether my reasoning is actually sound. Maybe I'm wrong and we can somehow make them interpretable in a sense that is actually relevant to verifying them.