3 ms·
I have thought for a while about "brain surgery" for deep neural nets, much like a permanent kind of dropout. The idea is that you black out a set of neurons/f
by bayesian_horse 8y ago
I have thought for a while about "brain surgery" for deep neural nets, much like a permanent kind of dropout.
The idea is that you black out a set of neurons/filters and then train for a short while to overcome the performance penalty. To find the "set" of blacked out cells you could use a genetic algorithm or something, gradually increasing the number of masks.
The last step would be rearranging the network such that the not-blacked-out cells are contiguous, but form smaller layers.
And I remember Hinton hinted at replacing multiple layers by one layer (or no layer), or big layers by smaller layers through retraining.
- unixhero 8y agoAnd then what would happen? What is the effect of this? Very deep an interesting thoughts.
- space_fountain 8y agoI think mostly faster training time. If you can work out connections that don't matter early in the process you could start ignoring them.
- bayesian_horse 8y agoMy idea was more about faster inference by reducing the required number of operations. It might not be worthwhile at all!
- _0ffh 8y agoWell, there are long standing network pruning techniques like Optimal Brain Damage (1990) [1] and Optimal Brain Surgeon [2]. Could you be more concrete about what you would be trying to do? [1] https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.7223 https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.... [2] https://papers.nips.cc/paper/647-second-order-derivatives-for-network-pruning-optimal-brain-surgeon.pdf https://papers.nips.cc/paper/647-second-order-derivatives-fo...
- bayesian_horse 8y agoI didn't invest any serious work in this idea, and didn't even research if somebody has come up with such things before.