3 ms·
Maybe I'm just not conversant enough in the field (I do ML "ops", not ML) but I'm surprised there aren't clever iterative ways of "reducing" the size of a model
by dnautics 6y ago
Maybe I'm just not conversant enough in the field (I do ML "ops", not ML) but I'm surprised there aren't clever iterative ways of "reducing" the size of a model, for example, rank order the average activation of each parameter in a reducible group across the dataset, find the weakest, set it to zero, identify new misses, retrain with enrichment in the dead samples, reduce the hyperparameter by one.
- TrainedMonkey 6y agoThere are: https://medium.com/syncedreview/mit-google-propose-automl-for-model-compression-810be5b007d4 https://medium.com/syncedreview/mit-google-propose-automl-fo... I went to a more recent talk by Google and results they handpicked to show were impressive.
- 317070 6y agoThere are, it's called network pruning. You generally can reduce network size 90-95% without loss of accuracy with these methods.
- ChrisRackauckas 6y agoYou "just" have to train a huge network first, which is the downside.