I really do like the practical challenge that Vapnik poses and based on what you wrote I support the agenda. In the face of the trial&error mentality prevalent

I really do like the practical challenge that Vapnik poses and based on what you wrote I support the agenda. In the face of the trial&error mentality prevalent in ML, it does strike me as a wise approach towards advancement. However, the minor technicality remains in my view: there exists a case where a sufficiently complex causal mechanism can be only asympotically identified from a given DGP. In that case, more data is better, and more data is perhaps even required - but also the model will be fully general and will not start to fit on the randomness in sample/DGP for this "mechanism". So indeed, we can theoretically construct situations where Vapnik's intuition does not hold. Then, "more data is better => model is fitting sampling error" is more of a heuristic - even if it is a good one! If the model manages to disentangle error from mechanism, then more data could still improve the model while retaining generality. Of course, one might propose that any causal mechanism will be identified "sufficiently well" in a finite sample, so there's a point where we have enough data. Perhaps this can even be proven, and my theoretical counterexample simply never exists. In that case, you'd be right: A model that keeps learning ad-infinum can not be causally identified. I am not aware of such a proof, but I'd not be surprised if it were the case. Or more practically speaking, one may be able to show that improvements to the causal estimates are, at some point, always inferior to the overfitting errors no matter the DGP/model. I would perhaps propose a reverse of the heuristic: We know a model is good, if it can demonstrate that more data does NOT influence its invariant (causally identified) parts. This will be the challenge to solve for DNNs, as the combination of model+ latent DGP necessarily makes this a matter of assumption. In classical models, we can formulate these assumptions mathematically e.g. as exclusion restrictions and the like. For DNNs, I think we do not have such results yet.