Microsoft research has a really nice paper showing that the NTK explanation is very unlikely: https://www.microsoft.com/en-us/research/blog/three-mysteries-in-d

Microsoft research has a really nice paper showing that the NTK explanation is very unlikely: https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation/ https://www.microsoft.com/en-us/research/blog/three-mysterie...