8 ms·

Ways to fix statistics

134 points by aqsalose 9y ago

aqsalose 9y ago
See also further discussion on Andrew Gelman's blog: http://andrewgelman.com/2017/11/28/five-ways-fix-statistics/ http://andrewgelman.com/2017/11/28/five-ways-fix-statistics/
schuetze 9y ago
Step 1: Become Bayesian. Seriously though, I am inclined towards approaches, such as pre-registration, which limit the number of researcher degrees of freedom in analysis. It's not necessarily that statistics are broken. It's that the system incentivizes researchers to break the assumptions underlying these statistical tests.
- analog31 9y ago
  What I wonder is, given a system with perverse incentives, won't people find a way to abuse Bayesian statistics?
  - timClicks 9y ago
    Probably.. but with Bayesian methods at your model/data updates its priors, rather than you effectively embedding your prior beliefs into your models via selectively choosing tests that support them.
    - dnautics 9y ago
      It's very easy to let that slip into post hoc justification of your priors.
      - analog31 9y ago
        In my view, "prior" may be a misnomer. There is nothing that I'm aware of in Bayes' theorem to suggest that you have to formulate your priors before gathering or analyzing your data. I would describe priors as constraints that are included in an analysis, to narrow the results based on additional information that you're aware of. Bayes' theorem mainly provides a framework for computing what happens when you do that.
        kgwgk 9y ago
        Maybe the prior doesn’t need to be formulated _before_ getting the data, but it needs to be _independent_ of the data.
  - lliiffee 9y ago
    I'm sure they would! But there's a pretty strong argument it's a step in the right direction. Actually, this link makes a more modest proposal that papers should report likelihoods rather than p-values. This avoids reported results results depending on priors (which perhaps we don't trust authors to choose well), though a reader can easily impose themselves if they want to. https://arbital.com/p/likelihoods_not_pvalues/ https://arbital.com/p/likelihoods_not_pvalues/ You still have the option to muck with things by choosing your hypothesis class in a bad way-- nothing can really replace publishing data!
- madhadron 9y ago
  Why become Bayesian? The rational actor theorems that drove that in the '60's turn out not to apply if you use a class of priors instead of a single prior, and single priors are overspecific to capture belief in most cases. Better to go all the way down to decision theory.
  - Bromskloss 9y ago
    Do you have some fuller description of, or some links to, the work you are referring to? Also, would decision theory replace Bayesian inference? Would it not rather _use_ it?
- rocqua 9y ago
  What informs your priors? That is the core issue with practical applications of Bayesian statistics. Especially because it is a totally new parameter that people can complain about. At best this leads to rampant bikeshedding, at worst we get prior-hacking as opposed to p-hacking.
basseq 9y ago
I consider myself analytically-minded, and statistics still gives me a headache. Most people understand basic relationships: "These numbers are different, but not meaningfully so" or, "These two elements are more strongly related than these other two." The problem is threefold: 1. Learning statistics is a tautology. Meaning: the best way to learn statistics is to already know statistics. Go google how to do basic significance calculations in, say, Excel. The first paragraph of the resulting articles will dive into t-test and twin-tales and P-values and on down the lines. Ask someone how to do a simple comparison, and they'll digress into null hypotheses and "what test did you decide to run". You'll probably see some nasty formulas and opining about which PhD-level approach is better. It's just opaque. 2. Tools are professional grade. Software needs to bridge the gap between users' intent and their ability to do the analysis. Statistics requires R (you have to know how to program and statistics), SAS ($$$), or another package that limits barrier to entry. Most people know how to ask the basic question, but no idea how to get the answer or interpret the results. 3. People who know statistics can't interpret results to laymen. I see this a lot in business: really bright folks build segmentation models and predictive models, you start talking about cohorts and medians... but that's where it ends. Your c-suite doesn't care about that: it cares about the so-what.
- btilly 9y ago
  The root problem is even simpler. The question that everyone wants from statistics is, "Is this true?" The question that data properly answers is, "Here is how to update your beliefs." Therefore every incentive exists to pose questions that can produce answers which are MISUNDERSTOOD as being answers to the question we wanted answered which we couldn't. And once a particular way of answering it has become popular, every incentive exists to push it to draw conclusions because conclusions are better than the alternative. That's why p-values are so popular and misused. That's why you can't keep people from p-hacking their way to something publishable. My preference is to not fight it, but accept it. Make null results publishable. Have businesses focus on decision procedures, not statistics. You can argue all day with things like https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html that people shouldn't misuse statistics. Or you can decide the maximum number of conversions that you'll spend, call that N, stop if one version gets sqrt(N) ahead or if you get to N without a clear answer. (In the latter case, go with the one ahead as the best available answer.) The second procedure has a rigorous statistical interpretation as "the best answer available with the effort we were willing to put in". But it isn't a definitive answer, and doesn't pretend to be. Complex conversations are avoided. Get an answer, go with it.
danieltillett 9y ago
While there are problems with the way statistics are used by scientists and others, the elephant in room is the incentive system. p-hacking and the like result from the need to get “exciting and publishable” results on a consistent basis or effectively be fired (the simple path is no “sexy” results -> no “high quality” papers -> no grant funding -> no job). A possible solution that I have not seen proposed (this is mostly likely my ignorance) is for journals to only accept/reject research proposals, not finished papers. The journal would publish the resulting paper no matter the outcome of the research.
- Vinnl 9y ago
  The problem is that journals have incentives too. And these unfortunately often align: sexy results -> more income.
  - danieltillett 9y ago
    This is true, but it is surprisingly weak as the income of a journal is not too closely related to its impact factor. My proposal would still mean that Nature would get the most sexy papers because it would get the most sexy proposals. A bigger problem would be journals choosing proposals from authors with a history of getting sexy results rather than on the quality of the proposal. This reliance on "track record" is why the grant system is so broken.
    - Vinnl 9y ago
      Isn't it? I mean, I know there's no hard data on it since most journal subscriptions are bought in bulk, but the journals are using their impact factor to sell those subscriptions (case in point: [1]). Likewise, if they charge publication fees, authors will be most willing to pay large sums of (their funders') money for journals with high impact factors. And do you really think Nature would still get sexy results if there are sexy proposals? Or would your proposal (which I do agree with, for the same reason the traditional publishers won't do it:) lead to better research but fewer "sexy" results? After all, journals with a high impact factor are also known to have more retractions. (The traditional) publishers are going to keep doing what is bringing in the most money - or at least, what they believe what will. At this point, they don't believe that your system would lead to higher income. What could change that if authors are no longer incentivised to chase high impact factor journals. But that means there will have to be better ways to evaluate them that will actually get adopted (i.e. not require evaluators to comb through every applicant's research [2]). This is a difficult problem. And yes, I realise I'm not coming up with a solution. The only ideas I have on that front have a really small chance of success, but it's a bit too much to elaborate on that here. (Although in time I will write about them at [3].) [1] https://mobile.twitter.com/simoxenham/status/235698457124425728/photo/1 https://mobile.twitter.com/simoxenham/status/235698457124425... [2] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea... [3] https://medium.com/flockademic https://medium.com/flockademic
MichailP 9y ago
Is there a calculus based approach to statistics? Like understanding linear regression simply as a minimization problem, but ALL THE WAY ie extending to other statistical techniques?
- fny 9y ago
  It's called probability theory, but that's not going to resolve these issues. If you're interested, there's a great 100-level course online from Joe Blitzstein at Harvard: https://projects.iq.harvard.edu/stat110/home https://projects.iq.harvard.edu/stat110/home
  - catnaroek 9y ago
    It's called “probability theory”, but is it actually probability theory? How can you have a course in “probability theory” that doesn't mention the Lebesgue integral even once?
    - rocqua 9y ago
      I took a course in "probability theory" that never mentioned Lebesgue integrals. Those came in measure and integration later. The course was about distributions, how product distributions interact, the central limit theorem, etc. That is, it was about turning stochastic models into predictions. Later we had a course "Statistics" that is about matching seen results to stochastic models.
      - catnaroek 9y ago
        > The course was about distributions, How can you study these without knowing the Lebesgue integral? > how product distributions interact How can you do this without the Fubini-Tonelli theorem? etc. etc. etc.
        rocqua 9y ago
        By talking about the operations without giving the rigorous backgrounds. Heck, you can do quite a bit with rieman integrals, and for e.g. binomial distributions you just need sums.
rvern 9y ago
Likelihood functions, p-values, and the replication crisis: https://arbital.com/p/likelihoods_not_pvalues/?l=4xx https://arbital.com/p/likelihoods_not_pvalues/?l=4xx.
jpfed 9y ago
One can imagine workflows that combine some of these solutions. Imagine pre-registering experiments and analyses... and then fellow scientists privately weigh in with their predictions for the results. These predictions can then be used to form a prior for a false-positive risk calculation.
stewbrew 9y ago
I think this is an interesting collection because most proposals acknowledge that statistics cannot be fixed by means of statistics alone but rather by taking into account the human factor. Bayesian statistics won't keep humans, who want to achieve something that's not primarily about statistics, from doing stupid things.
aisofteng 9y ago
>We need more observational studies and randomized trials — more epidemiology on how people collect, manipulate, analyse, communicate and consume data. I don't understand what is meant by "epidemiology" in this sentence.
jamez1 9y ago
> Worse, NHST is often taken to mean that any data can be used to decide between two inverse claims: either ‘an effect’ that posits a relationship between, say, a treatment and an outcome (typically the favoured hypothesis) or ‘no effect’ (defined as the null hypothesis). This is the whole problem, the rejection of the null hypothesis is not just a ritual to follow before you then accept the alternate. Fisher's core idea is that it only takes one piece of evidence to reject something, while it should take many to confirm something. So the only conclusion you should draw is about what you've rejected. You can't then take that and infer the acceptance of something else. It's a subtle but important distinction that is the whole reason for this process.

134 points by aqsalose 9y ago

See also further discussion on Andrew Gelman's blog: http://andrewgelman.com/2017/11/28/five-ways-fix-statistics/ http://andrewgelman.com/2017/11/28/five-ways-fix-statistics/

schuetze 9y ago

Step 1: Become Bayesian. Seriously though, I am inclined towards approaches, such as pre-registration, which limit the number of researcher degrees of freedom in analysis. It's not necessarily that statistics are broken. It's that the system incentivizes researchers to break the assumptions underlying these statistical tests.

analog31 9y ago
What I wonder is, given a system with perverse incentives, won't people find a way to abuse Bayesian statistics?
- timClicks 9y ago
  Probably.. but with Bayesian methods at your model/data updates its priors, rather than you effectively embedding your prior beliefs into your models via selectively choosing tests that support them.
  - dnautics 9y ago
    It's very easy to let that slip into post hoc justification of your priors.
    - analog31 9y ago
      In my view, "prior" may be a misnomer. There is nothing that I'm aware of in Bayes' theorem to suggest that you have to formulate your priors before gathering or analyzing your data. I would describe priors as constraints that are included in an analysis, to narrow the results based on additional information that you're aware of. Bayes' theorem mainly provides a framework for computing what happens when you do that.
      - kgwgk 9y ago
        Maybe the prior doesn’t need to be formulated _before_ getting the data, but it needs to be _independent_ of the data.
- lliiffee 9y ago
  I'm sure they would! But there's a pretty strong argument it's a step in the right direction. Actually, this link makes a more modest proposal that papers should report likelihoods rather than p-values. This avoids reported results results depending on priors (which perhaps we don't trust authors to choose well), though a reader can easily impose themselves if they want to. https://arbital.com/p/likelihoods_not_pvalues/ https://arbital.com/p/likelihoods_not_pvalues/ You still have the option to muck with things by choosing your hypothesis class in a bad way-- nothing can really replace publishing data!
madhadron 9y ago
Why become Bayesian? The rational actor theorems that drove that in the '60's turn out not to apply if you use a class of priors instead of a single prior, and single priors are overspecific to capture belief in most cases. Better to go all the way down to decision theory.
- Bromskloss 9y ago
  Do you have some fuller description of, or some links to, the work you are referring to? Also, would decision theory replace Bayesian inference? Would it not rather _use_ it?
rocqua 9y ago
What informs your priors? That is the core issue with practical applications of Bayesian statistics. Especially because it is a totally new parameter that people can complain about. At best this leads to rampant bikeshedding, at worst we get prior-hacking as opposed to p-hacking.

basseq 9y ago

I consider myself analytically-minded, and statistics still gives me a headache. Most people understand basic relationships: "These numbers are different, but not meaningfully so" or, "These two elements are more strongly related than these other two." The problem is threefold: 1. Learning statistics is a tautology. Meaning: the best way to learn statistics is to already know statistics. Go google how to do basic significance calculations in, say, Excel. The first paragraph of the resulting articles will dive into t-test and twin-tales and P-values and on down the lines. Ask someone how to do a simple comparison, and they'll digress into null hypotheses and "what test did you decide to run". You'll probably see some nasty formulas and opining about which PhD-level approach is better. It's just opaque. 2. Tools are professional grade. Software needs to bridge the gap between users' intent and their ability to do the analysis. Statistics requires R (you have to know how to program and statistics), SAS ($$$), or another package that limits barrier to entry. Most people know how to ask the basic question, but no idea how to get the answer or interpret the results. 3. People who know statistics can't interpret results to laymen. I see this a lot in business: really bright folks build segmentation models and predictive models, you start talking about cohorts and medians... but that's where it ends. Your c-suite doesn't care about that: it cares about the so-what.

btilly 9y ago
The root problem is even simpler. The question that everyone wants from statistics is, "Is this true?" The question that data properly answers is, "Here is how to update your beliefs." Therefore every incentive exists to pose questions that can produce answers which are MISUNDERSTOOD as being answers to the question we wanted answered which we couldn't. And once a particular way of answering it has become popular, every incentive exists to push it to draw conclusions because conclusions are better than the alternative. That's why p-values are so popular and misused. That's why you can't keep people from p-hacking their way to something publishable. My preference is to not fight it, but accept it. Make null results publishable. Have businesses focus on decision procedures, not statistics. You can argue all day with things like https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html that people shouldn't misuse statistics. Or you can decide the maximum number of conversions that you'll spend, call that N, stop if one version gets sqrt(N) ahead or if you get to N without a clear answer. (In the latter case, go with the one ahead as the best available answer.) The second procedure has a rigorous statistical interpretation as "the best answer available with the effort we were willing to put in". But it isn't a definitive answer, and doesn't pretend to be. Complex conversations are avoided. Get an answer, go with it.

danieltillett 9y ago

While there are problems with the way statistics are used by scientists and others, the elephant in room is the incentive system. p-hacking and the like result from the need to get “exciting and publishable” results on a consistent basis or effectively be fired (the simple path is no “sexy” results -> no “high quality” papers -> no grant funding -> no job). A possible solution that I have not seen proposed (this is mostly likely my ignorance) is for journals to only accept/reject research proposals, not finished papers. The journal would publish the resulting paper no matter the outcome of the research.

Vinnl 9y ago
The problem is that journals have incentives too. And these unfortunately often align: sexy results -> more income.
- danieltillett 9y ago
  This is true, but it is surprisingly weak as the income of a journal is not too closely related to its impact factor. My proposal would still mean that Nature would get the most sexy papers because it would get the most sexy proposals. A bigger problem would be journals choosing proposals from authors with a history of getting sexy results rather than on the quality of the proposal. This reliance on "track record" is why the grant system is so broken.
  - Vinnl 9y ago
    Isn't it? I mean, I know there's no hard data on it since most journal subscriptions are bought in bulk, but the journals are using their impact factor to sell those subscriptions (case in point: [1]). Likewise, if they charge publication fees, authors will be most willing to pay large sums of (their funders') money for journals with high impact factors. And do you really think Nature would still get sexy results if there are sexy proposals? Or would your proposal (which I do agree with, for the same reason the traditional publishers won't do it:) lead to better research but fewer "sexy" results? After all, journals with a high impact factor are also known to have more retractions. (The traditional) publishers are going to keep doing what is bringing in the most money - or at least, what they believe what will. At this point, they don't believe that your system would lead to higher income. What could change that if authors are no longer incentivised to chase high impact factor journals. But that means there will have to be better ways to evaluate them that will actually get adopted (i.e. not require evaluators to comb through every applicant's research [2]). This is a difficult problem. And yes, I realise I'm not coming up with a solution. The only ideas I have on that front have a really small chance of success, but it's a bit too much to elaborate on that here. (Although in time I will write about them at [3].) [1] https://mobile.twitter.com/simoxenham/status/235698457124425728/photo/1 https://mobile.twitter.com/simoxenham/status/235698457124425... [2] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea... [3] https://medium.com/flockademic https://medium.com/flockademic

MichailP 9y ago

Is there a calculus based approach to statistics? Like understanding linear regression simply as a minimization problem, but ALL THE WAY ie extending to other statistical techniques?

fny 9y ago
It's called probability theory, but that's not going to resolve these issues. If you're interested, there's a great 100-level course online from Joe Blitzstein at Harvard: https://projects.iq.harvard.edu/stat110/home https://projects.iq.harvard.edu/stat110/home
- catnaroek 9y ago
  It's called “probability theory”, but is it actually probability theory? How can you have a course in “probability theory” that doesn't mention the Lebesgue integral even once?
  - rocqua 9y ago
    I took a course in "probability theory" that never mentioned Lebesgue integrals. Those came in measure and integration later. The course was about distributions, how product distributions interact, the central limit theorem, etc. That is, it was about turning stochastic models into predictions. Later we had a course "Statistics" that is about matching seen results to stochastic models.
    - catnaroek 9y ago
      > The course was about distributions, How can you study these without knowing the Lebesgue integral? > how product distributions interact How can you do this without the Fubini-Tonelli theorem? etc. etc. etc.
      - rocqua 9y ago
        By talking about the operations without giving the rigorous backgrounds. Heck, you can do quite a bit with rieman integrals, and for e.g. binomial distributions you just need sums.

rvern 9y ago

Likelihood functions, p-values, and the replication crisis: https://arbital.com/p/likelihoods_not_pvalues/?l=4xx https://arbital.com/p/likelihoods_not_pvalues/?l=4xx.

jpfed 9y ago

One can imagine workflows that combine some of these solutions. Imagine pre-registering experiments and analyses... and then fellow scientists privately weigh in with their predictions for the results. These predictions can then be used to form a prior for a false-positive risk calculation.

stewbrew 9y ago

I think this is an interesting collection because most proposals acknowledge that statistics cannot be fixed by means of statistics alone but rather by taking into account the human factor. Bayesian statistics won't keep humans, who want to achieve something that's not primarily about statistics, from doing stupid things.

aisofteng 9y ago

>We need more observational studies and randomized trials — more epidemiology on how people collect, manipulate, analyse, communicate and consume data. I don't understand what is meant by "epidemiology" in this sentence.

jamez1 9y ago

> Worse, NHST is often taken to mean that any data can be used to decide between two inverse claims: either ‘an effect’ that posits a relationship between, say, a treatment and an outcome (typically the favoured hypothesis) or ‘no effect’ (defined as the null hypothesis). This is the whole problem, the rejection of the null hypothesis is not just a ritual to follow before you then accept the alternate. Fisher's core idea is that it only takes one piece of evidence to reject something, while it should take many to confirm something. So the only conclusion you should draw is about what you've rejected. You can't then take that and infer the acceptance of something else. It's a subtle but important distinction that is the whole reason for this process.