5 ms·

Maybe I'm just ignorant of the relevant statistical methods, but how do you suggest models be validated prospectively? All the model validation techniques I'm a

Maybe I'm just ignorant of the relevant statistical methods, but how do you suggest models be validated prospectively? All the model validation techniques I'm aware of come after the model is fit on the data (or a subset of it), like testing against a holdout set, or k-fold cross-validation, or checking R^2.

timr 5y ago
> testing against a holdout set Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it. It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.
- spekcular 5y ago
  What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout set collected in the future? I'm in total agreement that avoiding, e.g., overfitting is a good thing – but what work is prospectivity doing here? Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
  - timr 5y ago
    A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
    - spekcular 5y ago
      > A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
      - lazide 5y ago
        Not the poster, but happy to answer since I have a decent amount of experience in similar areas. Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking. You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not. That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data. It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
- teachingassist 5y ago
  > a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model. Since, you can check this model in the future - you can do it for yourself. And yet, you seem to be complaining about this publication? (I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)

timr 5y ago

> testing against a holdout set Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it. It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.

spekcular 5y ago
What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout set collected in the future? I'm in total agreement that avoiding, e.g., overfitting is a good thing – but what work is prospectivity doing here? Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
- timr 5y ago
  A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
  - spekcular 5y ago
    > A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
    - lazide 5y ago
      Not the poster, but happy to answer since I have a decent amount of experience in similar areas. Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking. You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not. That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data. It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
teachingassist 5y ago
> a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model. Since, you can check this model in the future - you can do it for yourself. And yet, you seem to be complaining about this publication? (I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)