3 ms·

If I run N different machine learning models over the same data, and each has some random error in fitting the objective function, then I pick the one which mat

If I run N different machine learning models over the same data, and each has some random error in fitting the objective function, then I pick the one which matches the validation data best, isn't there a danger of picking the one which was "luckiest" with the random errors? Presumably for large N that's a real problem? How do people account for that?

mjburgess 5y ago
https://en.wikipedia.org/wiki/Cross-validation_(statistics) https://en.wikipedia.org/wiki/Cross-validation_(statistics)
- fancyfredbot 5y ago
  Cross validation doesn't solve that problem. As the Wikipedia article says: "The variance of F* can be large.[26][27] For this reason, if two statistical procedures are compared based on the results of cross-validation, the procedure with the better estimated performance may not actually be the better of the two procedures (i.e. it may not have the better value of EF). Some progress has been made on constructing confidence intervals around cross-validation estimates,[26] but this is considered a difficult problem. "
  - mjburgess 5y ago
    Well the historical data {(x, y)...} is assumed to be distributed according to the true distribution, such that y = t(x) where t is the true function which maps x to y. Of course, in many situations, no such function exists (ie., there are genuinely ambigious xs, such that t(x) cannot produce a single y -- consider an ambigous cat/dog picture). If we sweep models f1,...fn across the validation set ... and choose max() of scores() of f1..fn on V, we get f*. Now if your issue is that f* might be an "unlucky draw", you're right. But there is no statistical way of fixing this -- if we know (via experiment, etc.) what the true distribution is, we can meaasure |f* - t| -- but if we knew this, we wouldnt bother finding f*. If you want a mechanism to mitigate these problems, there is one main one: the scientific method. To test whether f* poorly reflects t, go and do some experiemnts. If you can't, then you wont know. (Hence: there is no way of doing science via "mere statistics". It is the experimental conditions constructed by concept-laden, in-the-world, experimenters which are able to obtain the sequence of datasets needed to give confidence to any given model. ML is therefore not able to know anything, its "conclusions" enterily derivative-of, and limited-by, human experimentation. The intelligence occurs in the experimental design, when that's done, everything else is "stamp collecting").
    - fancyfredbot 5y ago
      Thank you for the detailed response. I think this is exactly what I worry about. If you were cynical enough and had enough datasets x1...xn and models f1...fn you would eventually a dataset on which a simple model performs well and be able to publish a paper like this. Even if the author here wasn't cynical enough to do that on purpose, many individually well meaning researchers looking for a nice result to publish effectively do the same thing!
oneoff786 5y ago
You have a test set that isn’t used during training, separate from the validation set. But in general this isn’t too much of a problem. For large N sizes the models will tend to converge on the same logic. It’s actually much more of a problem with small samples. Simpler models like linear regressions and decision trees could even be deterministic.

mjburgess 5y ago

https://en.wikipedia.org/wiki/Cross-validation_(statistics) https://en.wikipedia.org/wiki/Cross-validation_(statistics)

fancyfredbot 5y ago
Cross validation doesn't solve that problem. As the Wikipedia article says: "The variance of F* can be large.[26][27] For this reason, if two statistical procedures are compared based on the results of cross-validation, the procedure with the better estimated performance may not actually be the better of the two procedures (i.e. it may not have the better value of EF). Some progress has been made on constructing confidence intervals around cross-validation estimates,[26] but this is considered a difficult problem. "
- mjburgess 5y ago
  Well the historical data {(x, y)...} is assumed to be distributed according to the true distribution, such that y = t(x) where t is the true function which maps x to y. Of course, in many situations, no such function exists (ie., there are genuinely ambigious xs, such that t(x) cannot produce a single y -- consider an ambigous cat/dog picture). If we sweep models f1,...fn across the validation set ... and choose max() of scores() of f1..fn on V, we get f*. Now if your issue is that f* might be an "unlucky draw", you're right. But there is no statistical way of fixing this -- if we know (via experiment, etc.) what the true distribution is, we can meaasure |f* - t| -- but if we knew this, we wouldnt bother finding f*. If you want a mechanism to mitigate these problems, there is one main one: the scientific method. To test whether f* poorly reflects t, go and do some experiemnts. If you can't, then you wont know. (Hence: there is no way of doing science via "mere statistics". It is the experimental conditions constructed by concept-laden, in-the-world, experimenters which are able to obtain the sequence of datasets needed to give confidence to any given model. ML is therefore not able to know anything, its "conclusions" enterily derivative-of, and limited-by, human experimentation. The intelligence occurs in the experimental design, when that's done, everything else is "stamp collecting").
  - fancyfredbot 5y ago
    Thank you for the detailed response. I think this is exactly what I worry about. If you were cynical enough and had enough datasets x1...xn and models f1...fn you would eventually a dataset on which a simple model performs well and be able to publish a paper like this. Even if the author here wasn't cynical enough to do that on purpose, many individually well meaning researchers looking for a nice result to publish effectively do the same thing!

oneoff786 5y ago

You have a test set that isn’t used during training, separate from the validation set. But in general this isn’t too much of a problem. For large N sizes the models will tend to converge on the same logic. It’s actually much more of a problem with small samples. Simpler models like linear regressions and decision trees could even be deterministic.