3 ms·
As long as (stochastic) gradient descent and random initialization is the state of the art, more data will be better.
by twayt 5y ago
As long as (stochastic) gradient descent and random initialization is the state of the art, more data will be better.
- kd5bjo 5y agoRandom initialization is already starting to give way. One of the more promising approaches right now is pretraining with synthetic data to obtain “good” initial weights. Once that converges, you’ve got a network that internally recognizes interesting features which is a bettter-than-random starting point for training with real-world data. cf. https://arxiv.org/abs/2106.05963 https://arxiv.org/abs/2106.05963