5 ms·

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enou

I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage. Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Where is it written that our null and alternative hypothesis should conform to what is true, when our best ability to predict the future (as people) is based on our past experience? The world doesn't always behave predictably, and yet that seems to be baked into the assumption of every regression test ever run. Completely bonkers if you ask me.

madhadron 7y ago
p-values are two way decision procedures. For historical reasons we report the critical value of the type I error, because that was the piece people figured out before we got decision theory and the Neyman-Pearson lemma. All the null hypothesis vs other hypothesis stuff is distractions from the basic fact of choosing between two decisions based on data. I wrote an explanation for clinicians a while back that I people have had good luck with: http://madhadron.com/posts/2016-01-25-p_values_for_clinicians.html http://madhadron.com/posts/2016-01-25-p_values_for_clinician... > Likewise, we're supposed to assume that there is something magical about our prior assumptions? No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.
TrinaryWorksToo 7y ago
I'm interested in your perspective. What would be a better replacement for p?
- patientplatypus 7y ago
  Ultimately, I think what we are trying to do is find out if a result is "surprising", as in it doesn't do what we would expect or it would do what we would expect, which is based on an emotional point of view from the perspective of the tester. Much like in art, in the sciences there is much left up to the muse, although in the sciences the dedication to rigor leads more often than not to a disregard to emotion or just assume that all utterances are by definition rational. I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change. But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out. EDIT: To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use. I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument. But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question? I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.
Filligree 7y ago
> Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow. You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.
- wnoise 7y ago
  Well, you can report odds ratios (or their log).
  - Filligree 7y ago
    That's good advice in general, but doesn't really answer the question of "How likely is it?".
    - wnoise 7y ago
      Right, because you can't answer that without prior information. Log-odds is a step towards giving a function of prior information (but it is limited to the actual class of hypotheses investigated).

madhadron 7y ago

p-values are two way decision procedures. For historical reasons we report the critical value of the type I error, because that was the piece people figured out before we got decision theory and the Neyman-Pearson lemma. All the null hypothesis vs other hypothesis stuff is distractions from the basic fact of choosing between two decisions based on data. I wrote an explanation for clinicians a while back that I people have had good luck with: http://madhadron.com/posts/2016-01-25-p_values_for_clinicians.html http://madhadron.com/posts/2016-01-25-p_values_for_clinician... > Likewise, we're supposed to assume that there is something magical about our prior assumptions? No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.

TrinaryWorksToo 7y ago

I'm interested in your perspective. What would be a better replacement for p?

patientplatypus 7y ago
Ultimately, I think what we are trying to do is find out if a result is "surprising", as in it doesn't do what we would expect or it would do what we would expect, which is based on an emotional point of view from the perspective of the tester. Much like in art, in the sciences there is much left up to the muse, although in the sciences the dedication to rigor leads more often than not to a disregard to emotion or just assume that all utterances are by definition rational. I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change. But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out. EDIT: To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use. I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument. But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question? I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.

Filligree 7y ago

> Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow. You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.

wnoise 7y ago
Well, you can report odds ratios (or their log).
- Filligree 7y ago
  That's good advice in general, but doesn't really answer the question of "How likely is it?".
  - wnoise 7y ago
    Right, because you can't answer that without prior information. Log-odds is a step towards giving a function of prior information (but it is limited to the actual class of hypotheses investigated).