4 ms·
Probably a strong neural net only needs sparse connections to learn well. However, we simple humans cannot predict which sparse connections are important. There
by lostdog 4y ago
Probably a strong neural net only needs sparse connections to learn well. However, we simple humans cannot predict which sparse connections are important. Therefore, the net needs to learn which connections are important, but learning the connections means it needs to compute all of them during the training process, so the training process is slow. It's very challenging to break this cycle!
- polygamous_bat 4y agoWhat you are describing is known as the "lottery ticket hypothesis" [0] in the ML world -- it is a well-studied phenomenon! [0] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635
- tehnub 4y agoAnother article on this blog talks about it: https://lilianweng.github.io/posts/2019-03-14-overfit/#the-lottery-ticket-hypothesis https://lilianweng.github.io/posts/2019-03-14-overfit/#the-l...