4 ms·
Another random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of
by icyfox 3y ago
Another random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of the layer. In theory it's possible to feed actual datasets into these models as well, which could be another way to get a sparse model that's more acutely optimized towards one task. The original model produces the X activations on each layer, and these are used as the optimization criteria for the pruned version.
It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.
- CuriouslyC 3y agoStatisticians have been using L1 regularization to estimate sparse models for a while, it seems reasonable to assume that you could fine tune the model on a data set while also pushing weak parameters to zero in a natural way in this domain as well.
- ianbutler 3y agoI believe you are correct, I worked on a summer research project at NYU in 2018 based on https://arxiv.org/abs/1805.12185 https://arxiv.org/abs/1805.12185 As part of that project I constructed an API that took a small dataset and a model, launched a K8s pod and ran something like this from the paper: > The pruning defense works as follows: the defender exercises the DNN received from the attacker with clean inputs from the validation dataset, D_valid, and records the average activation of each neuron. The defender then iteratively prunes neurons from the DNN in increasing order of average activations and records the accuracy of the pruned network in each iteration. The defense terminates when the accuracy on the validation dataset drops below a pre-determined threshold. We note that pruning has been proposed in prior work for n Obviously this wasn't on transformers but the idea is similar.