4 ms·
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 This is in fact the
by sanxiyn 7y ago
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635
This is in fact the most interesting hypothesis on why neural networks work I have ever read.
- frequentnapper 7y agothanks for this. I wonder if the biological pruning process the brain uses works similarly.
- amelius 7y agoThis makes me wonder, if a lottery ticket network is roughly 10% of the size of the original network, then I would expect that statistically you'd have to train a randomly initialized verion of it 10 times to get the same result once. Is that true? Or is the actual topology of importance here?
- mdda 7y ago~No. Suppose that 5 of 50 channels in a particular network layer made up the 'lottery ticket' for that layer. The number of 'combos' of 5 channels that were trained at the same time is 50C5 (i.e. ~2million [0]). Whereas training 5C5 (=1) 10 times only gives you 10 chances to get the right 5 together. At least, that's one way to think about it. [0]: : https://www.mathway.com/popular-problems/Finite%20Math/601828 https://www.mathway.com/popular-problems/Finite%20Math/60182...
- bitforger 7y agoFollow up from ICLR really elucidates some of the questions raised by this: https://eng.uber.com/deconstructing-lottery-tickets/ https://eng.uber.com/deconstructing-lottery-tickets/ https://arxiv.org/abs/1905.01067 https://arxiv.org/abs/1905.01067