4 ms·
So the lottery is so unlikely it is not worth thinking about? Are there bounds on the possibilities? Can’t you just check a lot of values and hope for the best?
by mensetmanusman 6y ago
So the lottery is so unlikely it is not worth thinking about? Are there bounds on the possibilities? Can’t you just check a lot of values and hope for the best?
- pas 6y agoUmm, on the contrary. It's worth thinking about, but not to directly solve it, but to help the training process converge faster. So if we can increase the chances by many orders of magnitude, then it'll help a lot with training. Basically it's still an open question of exactly how and why the training process can get stuck. It's almost as if the network gets sufficiently strong ideas (Bayesian priors) it starts to discard new evidence (the error signal doesn't help). That's probably why a maximum entropy (random) state is useful to "bootstrap" those efficient subnetworks that then learn fast.