It is well known among practitioners that not all models have equal learning capacity and that additional data is wasted on those with lower capacity - if you p

It is well known among practitioners that not all models have equal learning capacity and that additional data is wasted on those with lower capacity - if you plot a performance metric vs dataset size, you can see it just saturates and stops increasing. The point where it saturates is a proxy for the model's learning capacity. One of the pros of very big models like as deep networks was that they never seem to saturate in this manner - more data always gave better results, because there are ways to increase the model's learning capacity. We've recently learned that deep nets work in deep overfitting mode, beyond the usual U-shaped regime of bias-variance tradeoff that can be seen in the empirical risk curve. See this post for a good overview: https://lilianweng.github.io/lil-log/2019/03/14/are-deep-neural-networks-dramatically-overfitted.html#modern-risk-curve-for-deep-learning https://lilianweng.github.io/lil-log/2019/03/14/are-deep-neu... It was a surprise to me that adding more data could lower a performance metric tough - that's counter-intuitive.