4 ms·

> Another useful property of the model is interpretability. Is this true? my understanding is the hard part about interpreting neural networks is that there ar

by dkdcio 1y ago

> Another useful property of the model is interpretability. Is this true? my understanding is the hard part about interpreting neural networks is that there are many many neurons, with many many interconnections, not that the activation function itself is not explainable. even with an explainable classifier, how do you explain trillions of them with deep layers of nested connections

bobmarleybiceps 1y ago
I've decided 100% of papers saying their modification of a neural network is interpretable are exaggerating.
- tpoacher 1y ago
  Personally, I'm looking forward to MNNs: Mansplainable Neural Networks.
abeppu 1y ago
I think the case for interpretability could have been made better, but in Figure 3 I think if you look at the middle "prototype" rows from the traditional vs Tversky layers, and scroll so you can't see the rows above, I think you could pick out mostly which Tversky prototype corresponds to each digit, but not which traditional/linear prototype corresponds to each digit. So I do think that's more interpretable in two ways: 1. You can look at specific representations in the model and "see" what they "mean" 2. This means you can give a high-level interpretation to a particular inference run: "X_i is a 7 because it's like this prototype that looks like a 7, and it has some features that only turn up in 7s" I do think complex models doing complex tasks will sometimes have extremely complex "explanations" which may not really communicate anything to a human, and so do not function as an explanation.
- sdenton4 1y ago
  It's wishful thinking. Neutral networks need to be over parameterized to find good solutions, meaning there is a surface of solutions. The optimization procedure tries to walk towards that surface as quickly as possible, and tend to find a low-energy point on the surface of solutions. In particular, a low energy solution isn't sparse, and therefore isn't interpretable.
  - c32c33429009ed6 1y ago
    Interesting; can you provide some references for this way of thinking?

by dkdcio 1y ago

bobmarleybiceps 1y ago

I've decided 100% of papers saying their modification of a neural network is interpretable are exaggerating.

tpoacher 1y ago
Personally, I'm looking forward to MNNs: Mansplainable Neural Networks.

abeppu 1y ago

I think the case for interpretability could have been made better, but in Figure 3 I think if you look at the middle "prototype" rows from the traditional vs Tversky layers, and scroll so you can't see the rows above, I think you could pick out mostly which Tversky prototype corresponds to each digit, but not which traditional/linear prototype corresponds to each digit. So I do think that's more interpretable in two ways: 1. You can look at specific representations in the model and "see" what they "mean" 2. This means you can give a high-level interpretation to a particular inference run: "X_i is a 7 because it's like this prototype that looks like a 7, and it has some features that only turn up in 7s" I do think complex models doing complex tasks will sometimes have extremely complex "explanations" which may not really communicate anything to a human, and so do not function as an explanation.

sdenton4 1y ago
It's wishful thinking. Neutral networks need to be over parameterized to find good solutions, meaning there is a surface of solutions. The optimization procedure tries to walk towards that surface as quickly as possible, and tend to find a low-energy point on the surface of solutions. In particular, a low energy solution isn't sparse, and therefore isn't interpretable.
- c32c33429009ed6 1y ago
  Interesting; can you provide some references for this way of thinking?