3 ms·

What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout se

What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout set collected in the future? I'm in total agreement that avoiding, e.g., overfitting is a good thing – but what work is prospectivity doing here? Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!

timr 5y ago
A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
- spekcular 5y ago
  > A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
  - lazide 5y ago
    Not the poster, but happy to answer since I have a decent amount of experience in similar areas. Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking. You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not. That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data. It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
    - timr 5y ago
      Yes, exactly. But I'd extend this one part: > Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future. Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.
      - spekcular 5y ago
        The point about peeking at the holdout set is fine, I guess. But then the slogan should be "models without a truly blinded holdout set are statistical fiction." Prospectivity doesn't do any work above and beyond blinding there. I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.

timr 5y ago

A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.

spekcular 5y ago
> A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
- lazide 5y ago
  Not the poster, but happy to answer since I have a decent amount of experience in similar areas. Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking. You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not. That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data. It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
  - timr 5y ago
    Yes, exactly. But I'd extend this one part: > Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future. Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.
    - spekcular 5y ago
      The point about peeking at the holdout set is fine, I guess. But then the slogan should be "models without a truly blinded holdout set are statistical fiction." Prospectivity doesn't do any work above and beyond blinding there. I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.