Right. What we're talking about here are tools. We can't understand or debug the electrons moving through the chip. But we build systems using languages and we

Right. What we're talking about here are tools. We can't understand or debug the electrons moving through the chip. But we build systems using languages and we build tools to translate execution back into those languages (Debuggers) Or tools in the execution to allow us to understand what happened later (Logging, metrics.) I think the key problem right now is that there are not too many models for which tracing the execution lets us know much about what's happening. A single decision tree. A linear model. But add ensembles to trees or layers to linear models and the ability to understand what's happening through logging disappears. In this sense I think we're essentially in the pre-assembly language state of machine learning. For most programming languages, we assume correctness as a given. If it produces the wrong result on correct inputs, that's really bad. With machine learning, we don't have perfect correctness. Aka accuracy. We focus so much on improving accuracy that we rarely consider the trade-off in debuggability or interpretability. I think this will have to change. Advances in NLP and image stuff from deep learning are very exciting. Ignoring the evil AI concerns, I think the best step will be trying to understand better why the models did what they did. I think it will be a while before we can just ask the model and get a reliable answer. In the meantime, I think we'll work on a kind of meta programming where we integrate models better with their broader context. An example might be sequence to sequence models that have a way to expose sentence fragments and documents in the training set that influenced a word choice. Or automatic labeling of clusters in image representation layers that have semantic meaning to a human. There's some interesting work already on debugging tools for image classification that I really liked. On the phone so I'll try to provide the link later. The trouble is it all seems very domain specific. Figuring out how do it in a way that's not so domain specific is i think a very interesting thing to work on.