5 ms·

Hey! I'm one of the authors, along with Alex and Ludwig. We're happy to answer any questions! :)

by colah3 9y ago

breckinloggins 9y ago
Many of the visualizations seem interpretable even though they are (essentially, I presume) projections of a very high dimensional space onto 2d position + 3d color space. Is there any research on projecting these visualizations into 6-dimensions as well (3d position + 3d color in VR) or even higher dimensions by substituting other sensory modalities for extra spacial or color dimensions?
- shancarter 9y ago
  I suspect the extra fidelity won't have much payoff for high dimensions considering the large cognitive overhead required for a person to make sense of it. Even a six dimensional approximation of a 1,000 dimensional space is quite crude, and we're pretty confused by spaces larger than three.
cs702 9y ago
As always, THANK YOU. You and your coauthors' attention to even the smallest details everywhere in this piece are evident. I've added it to my reading list, to read through it and look at the visualizations slowly :-) That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
- colah3 9y ago
  We're glad you enjoyed it! :D > do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN? That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)
  - cs702 9y ago
    Thanks. I asked the question in such an open-ended way just to see if you had any crazy ideas. It does sound like a hard problem. In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?
    - laughingman2 9y ago
      https://blog.acolyer.org/2017/11/01/deepxplore-automated-whitebox-testing-of-deep-learning-systems/ https://blog.acolyer.org/2017/11/01/deepxplore-automated-whi... Check this out
    - cma 9y ago
      Sounds like http://ufldl.stanford.edu/tutorial/unsupervised/SparseCoding/ http://ufldl.stanford.edu/tutorial/unsupervised/SparseCoding... ?
TheAlchemist 9y ago
No questions. Just a big thank you ! distill.pub is really a really amazing place to learn.
amitp 9y ago
I wanted to thank you not only for this new page but also for your blog and distill.pub, especially the research debt page!
- colah3 9y ago
  Our pleasure -- very glad our work is helpful. ;)
wibr 9y ago
How useful are visualizations like this for improving the performance of the model?
Scarblac 9y ago
Do you often have beautiful pictures like this on your screen? (a reason why I'm moving into scientific visualization and GIS instead of backend work...)
- colah3 9y ago
  Yep, neural network interpretability research involves quite a bit of staring a pretty pictures. It's a difficult job, but someone has to look at all those images. :P
pinouchon 9y ago
Hi. Thanks for the great article. Can you elaborate on what you mean by "directions in activation space" ? If I understand it right: You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct? Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf https://arxiv.org/pdf/1312.6199.pdf) has the following abstract: > First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks. This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810 https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much." Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf https://arxiv.org/pdf/1710.09412.pdf)

Hey! I'm one of the authors, along with Alex and Ludwig. We're happy to answer any questions! :)

by colah3 9y ago

Hey! I'm one of the authors, along with Alex and Ludwig. We're happy to answer any questions! :)

Many of the visualizations seem interpretable even though they are (essentially, I presume) projections of a very high dimensional space onto 2d position + 3d color space. Is there any research on projecting these visualizations into 6-dimensions as well (3d position + 3d color in VR) or even higher dimensions by substituting other sensory modalities for extra spacial or color dimensions?

shancarter 9y ago
I suspect the extra fidelity won't have much payoff for high dimensions considering the large cognitive overhead required for a person to make sense of it. Even a six dimensional approximation of a 1,000 dimensional space is quite crude, and we're pretty confused by spaces larger than three.

cs702 9y ago

As always, THANK YOU. You and your coauthors' attention to even the smallest details everywhere in this piece are evident. I've added it to my reading list, to read through it and look at the visualizations slowly :-) That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?

colah3 9y ago
We're glad you enjoyed it! :D > do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN? That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)
- cs702 9y ago
  Thanks. I asked the question in such an open-ended way just to see if you had any crazy ideas. It does sound like a hard problem. In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?
  - laughingman2 9y ago
    https://blog.acolyer.org/2017/11/01/deepxplore-automated-whitebox-testing-of-deep-learning-systems/ https://blog.acolyer.org/2017/11/01/deepxplore-automated-whi... Check this out
  - cma 9y ago
    Sounds like http://ufldl.stanford.edu/tutorial/unsupervised/SparseCoding/ http://ufldl.stanford.edu/tutorial/unsupervised/SparseCoding... ?

TheAlchemist 9y ago

No questions. Just a big thank you ! distill.pub is really a really amazing place to learn.

amitp 9y ago

I wanted to thank you not only for this new page but also for your blog and distill.pub, especially the research debt page!

colah3 9y ago
Our pleasure -- very glad our work is helpful. ;)

wibr 9y ago

How useful are visualizations like this for improving the performance of the model?

Scarblac 9y ago

Do you often have beautiful pictures like this on your screen? (a reason why I'm moving into scientific visualization and GIS instead of backend work...)

colah3 9y ago
Yep, neural network interpretability research involves quite a bit of staring a pretty pictures. It's a difficult job, but someone has to look at all those images. :P

pinouchon 9y ago

Hi. Thanks for the great article. Can you elaborate on what you mean by "directions in activation space" ? If I understand it right: You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct? Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf https://arxiv.org/pdf/1312.6199.pdf) has the following abstract: > First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks. This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810 https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much." Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf https://arxiv.org/pdf/1710.09412.pdf)