3 ms·

So, what does interpretable really mean? When you train a neural network, it is also not really a black box. You can exactly see what it does, and inspect all

So, what does interpretable really mean? When you train a neural network, it is also not really a black box. You can exactly see what it does, and inspect all weights and activations. You have a very clear logical reasoning why you end up with some decision. Only that this is maybe a huge decision rule when written out. When you would now take such symbolic or logic based approach and apply it on some real-world task, and depending how much freedom you leave to the training, the learned logic rules can look exactly the same as the logic rules when you write down the decision rule of the neural network. Maybe the logic rules derived this way will look shorter, simpler, more understandable. But this is not really studied here. And this is questionable. And maybe even not so easy to measure anyway. More interesting is the increased data efficiency by the proposed method. This is something you can measure. And any improvement here is good and speaks for the new method by itself. But this does not imply better interpretability.

teruakohatu 5y ago
> When you train a neural network, it is also not really a black box. You can exactly see what it does, and inspect all weights and activations. You have a very clear logical reasoning why you end up with some decision. Only that this is maybe a huge decision rule when written out. In some networks eg. CNN you may be able to visualize which of the inputs have the greatest impact on the output(s) but that is a far cry from understanding how the network got there in the first place. When a network gets something consistently wrong the answer is usually "more/better training data" not a deep understanding of why the network is wrong. This is in contrast to a simple decision tree, whose decisions can be understood by even a layperson with no statistics background. It is simple to go back and understand why the wrong decision was made.
aabaker99 5y ago
Can you inspect the weights in GPT-3 and tell me what does? The mathematics of neural networks leads to huge numbers of parameters with small contributions on the overal result. There's tons of work in ML to try and go in the opposite direction, LASSO being probably the most famous. Yes, you can inspect the weights. But compared to other models, neural networks are still a black box. Interpretable means a human being able to explain what the model does. LASSO gets you this by narrowing down on a small number of important parameters. Decision trees are interpretable. Even allowing for some groupings of neural network parameters like convolutional filters or certain network architecture choices that can characterize an entire layer, there's still too many parameters to explain.

teruakohatu 5y ago

> When you train a neural network, it is also not really a black box. You can exactly see what it does, and inspect all weights and activations. You have a very clear logical reasoning why you end up with some decision. Only that this is maybe a huge decision rule when written out. In some networks eg. CNN you may be able to visualize which of the inputs have the greatest impact on the output(s) but that is a far cry from understanding how the network got there in the first place. When a network gets something consistently wrong the answer is usually "more/better training data" not a deep understanding of why the network is wrong. This is in contrast to a simple decision tree, whose decisions can be understood by even a layperson with no statistics background. It is simple to go back and understand why the wrong decision was made.

aabaker99 5y ago

Can you inspect the weights in GPT-3 and tell me what does? The mathematics of neural networks leads to huge numbers of parameters with small contributions on the overal result. There's tons of work in ML to try and go in the opposite direction, LASSO being probably the most famous. Yes, you can inspect the weights. But compared to other models, neural networks are still a black box. Interpretable means a human being able to explain what the model does. LASSO gets you this by narrowing down on a small number of important parameters. Decision trees are interpretable. Even allowing for some groupings of neural network parameters like convolutional filters or certain network architecture choices that can characterize an entire layer, there's still too many parameters to explain.