3 ms·
Removing dead neurons is another way of reducing model size I’m assuming
by thatxliner 3y ago
Removing dead neurons is another way of reducing model size I’m assuming
- thomasahle 3y agoOr maybe one should keep track of "neuron activation frequency" during training. This would not a lot of extra parameters, since it's per neuron, not per weight. Then every epoch, or so, we'd "reinitialize" the dead neurons. This is similar to K-Means algorithms that reinitialize cluster centers that have very few assigned points.