Right, once we start heavily hyper-parameterizing our models, there's the potential to overfit the hyperparameters. I think most people agree that has already

Right, once we start heavily hyper-parameterizing our models, there's the potential to overfit the hyperparameters. I think most people agree that has already happened on common datasets such as MNIST. However, the current method of avoiding hyperparameters is just to have very few of them. This is kind of like avoiding overfitting of parameters by only having 10 parameters in your model - barbaric! That being said, for hyper-gradients to really be useful, someone needs to develop hyperparameter optimization schemes such as BayesOpt that can condition on gradient information, to allow us to try different hyperparameter settings in parallel. I know at least one group is working on this, but as far as I know it's not ready for prime time yet.