Well the historical data {(x, y)...} is assumed to be distributed according to the true distribution, such that y = t(x) where t is the true function which maps

Well the historical data {(x, y)...} is assumed to be distributed according to the true distribution, such that y = t(x) where t is the true function which maps x to y. Of course, in many situations, no such function exists (ie., there are genuinely ambigious xs, such that t(x) cannot produce a single y -- consider an ambigous cat/dog picture). If we sweep models f1,...fn across the validation set ... and choose max() of scores() of f1..fn on V, we get f*. Now if your issue is that f* might be an "unlucky draw", you're right. But there is no statistical way of fixing this -- if we know (via experiment, etc.) what the true distribution is, we can meaasure |f* - t| -- but if we knew this, we wouldnt bother finding f*. If you want a mechanism to mitigate these problems, there is one main one: the scientific method. To test whether f* poorly reflects t, go and do some experiemnts. If you can't, then you wont know. (Hence: there is no way of doing science via "mere statistics". It is the experimental conditions constructed by concept-laden, in-the-world, experimenters which are able to obtain the sequence of datasets needed to give confidence to any given model. ML is therefore not able to know anything, its "conclusions" enterily derivative-of, and limited-by, human experimentation. The intelligence occurs in the experimental design, when that's done, everything else is "stamp collecting").