4 ms·

Can someone give more technical detail on what they are showing with the "neurons"? They say "Each neuron is represented by a feature visualization with a huma

Can someone give more technical detail on what they are showing with the "neurons"? They say "Each neuron is represented by a feature visualization with a human-chosen concept labels to help quickly provide a sense of each neuron", and these neurons are selected from the final layer. I don't think I understand this.

the8472 6y ago
Start with random input, then incrementally optimize the input to maximize the activation of one of the nodes in the graph, the neuron. The visualization is one of those inputs that hit a maximum.
colah3 6y ago
Feature visualizations are described in this article: https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/ The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ https://distill.pub/2021/multimodal-neurons/ ).
- nl 6y ago
  Worth noting that Chris Olah (who wrote this comment) has led much of the interesting work in making feature visualisations useful. If you look for "Visualizing Neural Networks" on his page https://colah.github.io/ https://colah.github.io/ you'll find lots of other interesting links in this area.
- iujjkfjdkkdkf 6y ago
  Thank you! Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum. I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.
- suifbwish 6y ago
  Quick unrelated question as you seem to be a subject expert. Are there currently any neural network models that deal with multiple separate networks that occasionally trade or share nodes? It might be useful for modeling a system where members of an organization may leave and join another organization such as a business or a church
ben_w 6y ago
During the initial research into multi-layer neural networks, it appeared that only the input and output layers had any human-comprehendible meaning; anything else would be an indecipherable vector of how much weight each item on the previous layer should have, e.g. 0.1*pixel[0] + 0.004*pixel[2] + … 0.4*pixel[1024] In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”. In this case, the researchers did this to the CLIP network and found it was much easier than usual for humans to come up with sensible names for the type of thing any given neuron would recognise. You can also demonstrate what a neuron does by running the maths in reverse to get an input which would activate that neuron more strongly than any other input; in this case an image.
- Swizec 6y ago
  > In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”. Isn’t this also how neuroscientists identified brain regions? I guess it makes sense this would happen but it’s curious that we’ve been able to build a blackbox algorithm/technique that’s as difficult to reverse engineer as real brains.

the8472 6y ago

Start with random input, then incrementally optimize the input to maximize the activation of one of the nodes in the graph, the neuron. The visualization is one of those inputs that hit a maximum.

colah3 6y ago

Feature visualizations are described in this article: https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/ The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ https://distill.pub/2021/multimodal-neurons/ ).

nl 6y ago
Worth noting that Chris Olah (who wrote this comment) has led much of the interesting work in making feature visualisations useful. If you look for "Visualizing Neural Networks" on his page https://colah.github.io/ https://colah.github.io/ you'll find lots of other interesting links in this area.
iujjkfjdkkdkf 6y ago
Thank you! Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum. I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.
suifbwish 6y ago
Quick unrelated question as you seem to be a subject expert. Are there currently any neural network models that deal with multiple separate networks that occasionally trade or share nodes? It might be useful for modeling a system where members of an organization may leave and join another organization such as a business or a church

ben_w 6y ago

During the initial research into multi-layer neural networks, it appeared that only the input and output layers had any human-comprehendible meaning; anything else would be an indecipherable vector of how much weight each item on the previous layer should have, e.g. 0.1*pixel[0] + 0.004*pixel[2] + … 0.4*pixel[1024] In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”. In this case, the researchers did this to the CLIP network and found it was much easier than usual for humans to come up with sensible names for the type of thing any given neuron would recognise. You can also demonstrate what a neuron does by running the maths in reverse to get an input which would activate that neuron more strongly than any other input; in this case an image.

Swizec 6y ago
> In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”. Isn’t this also how neuroscientists identified brain regions? I guess it makes sense this would happen but it’s curious that we’ve been able to build a blackbox algorithm/technique that’s as difficult to reverse engineer as real brains.