5 ms·
https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531 Modern applications of small networks regularly reduce sizes from larger state-of-the-art net
by kajecounterhack 10y ago
https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531
Modern applications of small networks regularly reduce sizes from larger state-of-the-art networks using distillation. Distillation compacts neural networks while affecting accuracy minimally.
Instead of pruning directly from the large network, just learn how it generalizes. Takes fewer nodes / overall operations (Multiplications / Additions).
- Vitrified 10y agoNow that is interesting. I hadn't realized methods to combine trained networks so efficiently were already readily available.
- kajecounterhack 10y agoCertain companies use these methods to make state of the art neural nets work on your phones :) Also "combine" might not be the right word, since it's really transfer learning. "Distill" is really a descriptive verb. Maybe my original wording was confusing; I shouldn't have said "distillation compacts" -- distillation is a process by which you can create a more compact version of a complex neural net.