There's some super-interesting recent empirical work from MS research that makes this perspective less likely to be a useful one. In short, the behavior of neur

There's some super-interesting recent empirical work from MS research that makes this perspective less likely to be a useful one. In short, the behavior of neural networks in a few regimes makes this interpretation of them less likely, specifically: ensembling the same network trained multiple times separately improves results compared to training multiple networks concurrently, knowledge distillation works, and self-distillation works. The blog post[1] has a good intuitive explanation of the matter and a link to the paper. [1] https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation/ https://www.microsoft.com/en-us/research/blog/three-mysterie...