I basically share your sentiment. However, Greg Yangs work seems to have produced something of direct practical benefit for training large neural nets, based on

I basically share your sentiment. However, Greg Yangs work seems to have produced something of direct practical benefit for training large neural nets, based on the NTK literature. µ-Parametrization is apparently very useful in real world practice.