3 ms·

Crude methods are fine. Fantastic models are made with linear regression. No matter how crude your methods, if you don't validate your model prospectively, it

by timr 6y ago

Crude methods are fine. Fantastic models are made with linear regression. No matter how crude your methods, if you don't validate your model prospectively, it is a mathematical fairy tale.

spekcular 6y ago
Maybe I'm just ignorant of the relevant statistical methods, but how do you suggest models be validated prospectively? All the model validation techniques I'm aware of come after the model is fit on the data (or a subset of it), like testing against a holdout set, or k-fold cross-validation, or checking R^2.
- timr 6y ago
  > testing against a holdout set Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it. It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.
  - spekcular 6y ago
    What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout set collected in the future? I'm in total agreement that avoiding, e.g., overfitting is a good thing – but what work is prospectivity doing here? Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
    - timr 6y ago
      A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
      - spekcular 6y ago
        > A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
        6y ago
  - teachingassist 6y ago
    > a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model. Since, you can check this model in the future - you can do it for yourself. And yet, you seem to be complaining about this publication? (I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)

by timr 6y ago

Crude methods are fine. Fantastic models are made with linear regression. No matter how crude your methods, if you don't validate your model prospectively, it is a mathematical fairy tale.

spekcular 6y ago

Maybe I'm just ignorant of the relevant statistical methods, but how do you suggest models be validated prospectively? All the model validation techniques I'm aware of come after the model is fit on the data (or a subset of it), like testing against a holdout set, or k-fold cross-validation, or checking R^2.

timr 6y ago
> testing against a holdout set Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it. It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.
- spekcular 6y ago
  What is the mathematical difference between testing against a holdout set of data collected at the same time as the data used to fit the model, and a holdout set collected in the future? I'm in total agreement that avoiding, e.g., overfitting is a good thing – but what work is prospectivity doing here? Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
  - timr 6y ago
    A prospective test is blind. You don't know what the data looks like, so you can't cheat. > Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard! It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
    - spekcular 6y ago
      > A prospective test is blind. You don't know what the data looks like, so you can't cheat. Suppose I blind myself to the holdout set. What's the difference? > It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs. What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not? Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model? To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do. > Setting this aside: TFA did nothing you're talking about. Let's be clear about that. Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
      - 6y ago
- teachingassist 6y ago
  > a holdout set of prospective data, where you actually predict the future, blindly, and see how you do. According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model. Since, you can check this model in the future - you can do it for yourself. And yet, you seem to be complaining about this publication? (I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)