3 ms·

Most models have a purely fixed architecture: i.e. if you train three layers of six feed-forward neurons a piece, your model and the training data it learns fro

Most models have a purely fixed architecture: i.e. if you train three layers of six feed-forward neurons a piece, your model and the training data it learns from will just fit within that architecture to the extent that gradient descent can force it to fit. There is no mechanism to say "oh these neurons will never activate, let's prune them", or "we'd have much better loss if we added a layer here". In the dark times before PyTorch there was an idea called NEAT: "neuroevolution of augmenting topologies", which tried to use a genetic algorithm (i.e. testing a bunch of slightly modified solutions for loss) to discover both optimal weights and optimal network structure. I don't think this gets used all that often, if at all. I hear stories every few years about how many models have unused neurons that can be pruned or how hyperparameter selection is a pain in the ass, but nothing about automating the pruning and parameter selection in such a way as to efficiently use the whole model. I'm not sure it's necessary anyway.

tysam_and 3y ago
DL researcher here, it's also hard I think because experimentally many of us have noted (also some research on this) that there's a critical early phase to learning that conditions the network a certain way, and adding and/or removing layers later seems to actually be quite difficult (removing easier than adding, by far, esp w/ something like variational dropout [yes, i cite the old deep magicks]: https://arxiv.org/abs/1701.05369 https://arxiv.org/abs/1701.05369)
- hiddencost 3y ago
  I had success doubling the width of a layer by copying, adding epsilon noise and then fine tuning. Transform from [X] to [X X+epsilon]. In that case I was trying to fuse two signals that had fairly similar characteristics.
  - tysam_and 3y ago
    Yes, unfortunately I have yet to beat that technique in wallclock convergence speed vs just using the larger network from the start. :'(((( Whoever figures out how to clearly and effectively do it consistently faster than a 'full network from the start' version will open up an entirely new subfield of neural network training efficiency. :'))))
sdenton4 3y ago
There's a huge amount of work on model pruning, especially with an eye towards model reduction for on-device deployments. I've done a bunch of work in this space focused on getting speech synthesis working in real time on phones. It works and can be automated. There's a lot of nuance though. What has typically worked best are variants on iterative magnitude weight pruning rather than activation pruning. This can often get rid of 90% of the weights with minimal impact on quality. Structured pruning lets you remove blocks of weights while retaining good SIMD utilization... and so on.
radarsat1 3y ago
> There is no mechanism to say "oh these neurons will never activate, let's prune them" Not at training time. But after training you can evaluate which activations rarely fire and simply remove their columns from the weight matrix. Or you could re-randomize them and continue training.
- dTal 3y ago
  A neuron which rarely fires may be responding to a rare feature, no?
  - radarsat1 3y ago
    Absolutely. Pruning can decrease overall accuracy. The challenge is to prune as aggressively as possible while keeping accuracy as high as possible. It's certainly not an operation that is expected to have no effect. It should be noted that there are methods for training the network to encourage these prunable neurons, eg sparsity penalties.

tysam_and 3y ago

DL researcher here, it's also hard I think because experimentally many of us have noted (also some research on this) that there's a critical early phase to learning that conditions the network a certain way, and adding and/or removing layers later seems to actually be quite difficult (removing easier than adding, by far, esp w/ something like variational dropout [yes, i cite the old deep magicks]: https://arxiv.org/abs/1701.05369 https://arxiv.org/abs/1701.05369)

hiddencost 3y ago
I had success doubling the width of a layer by copying, adding epsilon noise and then fine tuning. Transform from [X] to [X X+epsilon]. In that case I was trying to fuse two signals that had fairly similar characteristics.
- tysam_and 3y ago
  Yes, unfortunately I have yet to beat that technique in wallclock convergence speed vs just using the larger network from the start. :'(((( Whoever figures out how to clearly and effectively do it consistently faster than a 'full network from the start' version will open up an entirely new subfield of neural network training efficiency. :'))))

sdenton4 3y ago

There's a huge amount of work on model pruning, especially with an eye towards model reduction for on-device deployments. I've done a bunch of work in this space focused on getting speech synthesis working in real time on phones. It works and can be automated. There's a lot of nuance though. What has typically worked best are variants on iterative magnitude weight pruning rather than activation pruning. This can often get rid of 90% of the weights with minimal impact on quality. Structured pruning lets you remove blocks of weights while retaining good SIMD utilization... and so on.

radarsat1 3y ago

> There is no mechanism to say "oh these neurons will never activate, let's prune them" Not at training time. But after training you can evaluate which activations rarely fire and simply remove their columns from the weight matrix. Or you could re-randomize them and continue training.

dTal 3y ago
A neuron which rarely fires may be responding to a rare feature, no?
- radarsat1 3y ago
  Absolutely. Pruning can decrease overall accuracy. The challenge is to prune as aggressively as possible while keeping accuracy as high as possible. It's certainly not an operation that is expected to have no effect. It should be noted that there are methods for training the network to encourage these prunable neurons, eg sparsity penalties.