Neural networks get a bad rap for being "black boxes" and "uninterpretable". But any decent machine learned model is incredibly difficult for people to understa

Neural networks get a bad rap for being "black boxes" and "uninterpretable". But any decent machine learned model is incredibly difficult for people to understand. There was a machine learning system specifically designed to produce interpretable models. It's called Eureqa, and it does symbolic regression. It finds the simplest possible mathematical expression that fits the data. Instead of millions of neural weights, you get a nice simple equation. How could not not be interpretable? But in any nontrivial case, it's still impossible to decipher. "Why is there a sine there? Why on Earth is it using mod? What could this constant possibly mean?" One biologist used it on some data he had been struggling with. And it found a simple equation that fit the data perfectly, which is what he had been looking for. But he couldn't publish it, because he couldn't possibly explain why it worked or how to derive it. You can't just publish a random equation with no understanding. I think the best method of understanding our models, is not going to come from making simpler models. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us. I suspect most of what NNs do isn't complicated in an absolute sense. Its just time consuming to go through the data and work it out. (Maybe we could outsource it to something like mechanical turk... "What do these images have in common?") There is a huge amount of effort put into making more accurate models, but much less into trying to interpret them. I think this is a huge mistake. Understanding a model lets you see it's weaknesses. The things that it can't learn. The mistakes it makes.