3 ms·
> There is no mechanism to say "oh these neurons will never activate, let's prune them" Not at training time. But after training you can evaluate which activat
by radarsat1 3y ago
> There is no mechanism to say "oh these neurons will never activate, let's prune them"
Not at training time. But after training you can evaluate which activations rarely fire and simply remove their columns from the weight matrix.
Or you could re-randomize them and continue training.
- dTal 3y agoA neuron which rarely fires may be responding to a rare feature, no?
- radarsat1 3y agoAbsolutely. Pruning can decrease overall accuracy. The challenge is to prune as aggressively as possible while keeping accuracy as high as possible. It's certainly not an operation that is expected to have no effect. It should be noted that there are methods for training the network to encourage these prunable neurons, eg sparsity penalties.