Random initialization is already starting to give way. One of the more promising approaches right now is pretraining with synthetic data to obtain “good” initi

Random initialization is already starting to give way. One of the more promising approaches right now is pretraining with synthetic data to obtain “good” initial weights. Once that converges, you’ve got a network that internally recognizes interesting features which is a bettter-than-random starting point for training with real-world data. cf. https://arxiv.org/abs/2106.05963 https://arxiv.org/abs/2106.05963