4 ms·
High-Low Frequency Detectors
- mbil 6y agoThe "Universality" section is especially interesting. They found these high-low frequency neurons in a number of different network architectures, and at around the same network depth in each case. I wonder how this insight could be useful. Can you train a network layer by layer and then connect the pre-trained layers? Like maybe train a "high-low frequency detector" layer on generated inputs of high-low frequency images, and then somehow put it 1/3 of the way into a network, maybe one whose other layers have been trained similarly to recognize other features.
- tingletech 6y agosidenote: the journal uses github for peer review https://distill.pub/guide/ https://distill.pub/guide/
- ludwigschubert 6y agoYeah! This particular article is part of a Thread and so didn’t go through peer review. Here’s an example of peer review on GitHub from an older article: https://github.com/distillpub/post--activation-atlas/issues?q=is%3Aissue+label%3Apeer-review https://github.com/distillpub/post--activation-atlas/issues?... In general on the Distill front page there are “peer-review” badges that link to the GitHub issues containing the reviews.
- ludwigschubert 6y agoCool to see this upvoted here, though I will say this is fairly deep down the weeds — if you’re new to the topic of analyzing trained neural networks neuron by neuron, I’d suggest starting with our introductory article: https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/