3 ms·

This is great. I think a good alternative interpretation of the findings that I haven't seen mentioned is through the lens of information theory. If a network h

This is great. I think a good alternative interpretation of the findings that I haven't seen mentioned is through the lens of information theory. If a network has generalized well, and each neuron is firing seemingly at random, then that sounds like the neurons of the network all have very high entropy. If you have neurons that fire only on certain inputs, then those neurons have low entropy (ie. you can accurately predict when they will fire, which is the definition of low entropy). So those neurons are less efficient at providing you with information. If you assume that the goal of the network is to maximize entropy, then I kind of would have thought that low entropy neurons would be the ones you'd want to delete first. So the fact that they're saying they have the same effect as more random ones is interesting. Or that there's another way I need to be thinking about how entropy can be measured for a given neuron... But I think that lens is a really good one for conceptualizing information flow through the network.

credit_guy 9y ago
I don't have a good intuition of neurons in terms of entropy, but I disagree with the conclusion they draw from their interpretability/importance graph - "Surprisingly, we found that there was little relationship between selectivity and importance." While highly selective neurons are not more or less important than generic neurons, the highly important neurons are always very non-selective. I think you call them high entropy neurons? "So the fact that they're saying they have the same effect as more random ones is interesting". Yes, they are saying that, but their graph doesn't show that. I think their graph supports your intuition, that if you ever plan to delete neurons from a NN, while minimizing the degradation, you should start with the low entropy neurons first.
- vannevar 9y ago
  While highly selective neurons are not more or less important than generic neurons, the highly important neurons are always very non-selective. This may simply reflect the fact that "selective" neurons are relatively rare. The chart in the article implies that there are equal numbers, but it's not clear whether the X-axis is linear or not, or where the threshold is for a selective neuron vs a non-selective one. If 95% of all neurons are non-selective, then the statement above is not surprising.

credit_guy 9y ago

I don't have a good intuition of neurons in terms of entropy, but I disagree with the conclusion they draw from their interpretability/importance graph - "Surprisingly, we found that there was little relationship between selectivity and importance." While highly selective neurons are not more or less important than generic neurons, the highly important neurons are always very non-selective. I think you call them high entropy neurons? "So the fact that they're saying they have the same effect as more random ones is interesting". Yes, they are saying that, but their graph doesn't show that. I think their graph supports your intuition, that if you ever plan to delete neurons from a NN, while minimizing the degradation, you should start with the low entropy neurons first.

vannevar 9y ago
While highly selective neurons are not more or less important than generic neurons, the highly important neurons are always very non-selective. This may simply reflect the fact that "selective" neurons are relatively rare. The chart in the article implies that there are equal numbers, but it's not clear whether the X-axis is linear or not, or where the threshold is for a selective neuron vs a non-selective one. If 95% of all neurons are non-selective, then the statement above is not surprising.