Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lea

Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lead to overfitting. Clearly there is a gap between out theoretical understanding and practice. Check out a recent paper that explores exactly this questions: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530