5 ms·
I'd recommend anyone interested in the NTK and its limitations to check out Greg Yang's work [1] and the recent book by Roberts, Yaida, and Hanin [2]. There's c
by r-zip 5y ago
I'd recommend anyone interested in the NTK and its limitations to check out Greg Yang's work [1] and the recent book by Roberts, Yaida, and Hanin [2]. There's clearly a lot more work to be done on neural network scaling limits; the NTK is overly simplistic (though useful).
1: https://www.microsoft.com/en-us/research/people/gregyang/ https://www.microsoft.com/en-us/research/people/gregyang/
2: https://arxiv.org/abs/2106.10165 https://arxiv.org/abs/2106.10165
- scythmic_waves 5y agoAgreed, Greg Yang's work is really interesting. Here's a reddit thread by Yang discussing his research into why NNs are not Kernel Machines [1]. [1] https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_wide_neural_networks_are_feature_learners_not/ https://www.reddit.com/r/MachineLearning/comments/k8h01q/r_w...