6 ms·

Thanks for sharing this. In general, it's a real shame how few people know how to interpret and use P values correctly. We work with a lot of businesses who ask

by cl42 12y ago

Thanks for sharing this. In general, it's a real shame how few people know how to interpret and use P values correctly. We work with a lot of businesses who ask us to compare populations (e.g., through A/B tests) and people either (a) don't care about the significance between population differences, or (b) are irrationally attached to P values. Case in point #1: debating whether a P value of 0.051 versus 0.049 is a major difference in the significance test. Case in point #2: a P value of <0.001 but with extremely low differences in means between populations. With enough data, everything is significant! End rant. :)

return0 12y ago
Amen for #2! It's like the entire field of biology is an endless quest for low p-values, regardless if the hypothesis is even interesting. It's like people are not interested to think, they just want to publish something significantly differnt.
- cl42 12y ago
  haha, Sociology as well -- especially now that the web provides huge amounts of behavioral data. I much prefer how machine learning folks tend to approach predictive accuracy, though I guess that's not quite the same as understanding relationships between specific variables while controlling for others.
  - sdenton4 12y ago
    There is a notion of 'feature importance,' which especially comes up in decision trees and random forests, giving a notion of how much a particular feature contributes to the overall prediction. It seems like combining predictive power with feature importance would be an interesting alternate route to demonstrating important correlations. (For example, maybe a model predicts lung cancer with 90% precision, and 'is_smoker' has a 80% feature importance.) Of course, these importances depend a lot on the other features used by the model! If you include a lot of junk features and/or exclude other important features, the importance of your pet feature will shoot up.
    - cl42 12y ago
      Hmm, interesting -- I never considered the idea of including junk features to bias model's preferences of whatever theoretically ambiguous idea you're trying to promote. That's actually brilliant.
      - mbq 12y ago
        Shameless plug; I'm a co-author of a method that leverages adding artificial junk features and removing original ones that are likely nonsense to approximate the set of all features that are relevant to the problem (rather than standard make best model, which may be pretty deceiving). https://m2.icm.edu.pl/boruta https://m2.icm.edu.pl/boruta
- shasta 12y ago
  Shoes reduce cholesterol.
  - cl42 12y ago
    And aren't ice cream sales dependent on murder rates?
    - copperx 12y ago
      Did you just discredit Freakonomics?
- Sharlin 12y ago
  > they just want to publish something What's going on in the academia is a classical example of perverse incentives. If your career depends on churning out soul-crushingly bad or insignificant papers that fulfill some superficial criteria of "scienceness" then that's what you'll do. It's dangerously close to cargo cult science.
conistonwater 12y ago
> are irrationally attached to P values This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not. I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome. It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.
- wodenokoto 12y ago
  > so they insist on following some strict rules, whether they make sense or not. I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data. As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005
  - RA_Fisher 12y ago
    Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
    - cgearhart 12y ago
      > The reason is that the p-value essentially tells you how repeatable your analysis will be Your statement contradicts the introductory paragraph of the article: > "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..." The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.
Scoundreller 12y ago
One of my favourite go-to phrases in health care is: "Statistically significant, but a clinically insignificant difference."
- cl42 12y ago
  I love it! That's actually a really good way of looking at it, right? Unfortunately for researchers, I'd argue that "Statistically significant, but a clinically insignificant difference." is equivalent to "Theoretically useless, and technically irrelevant." :)
- 6502nerdface 12y ago
  In quant finance, my colleagues and I say, "OK, it's statistically significant, but is it economically significant?" I love that we have parallel sayings in totally different fields.
- remarkEon 12y ago
  Right, you're describing statistical significance as it relates to practical significance.
joshdick 12y ago
Addressing that first point is "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant" by Andrew Gelman and Hal Stern. PDF link: http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf http://www.stat.columbia.edu/~gelman/research/published/sign...
- stdbrouw 12y ago
  Nitpick: while that paper does mention the basic idea that there's no significant difference between 0.051 and 0.049, it's more about another common mistake: testing the difference between two groups by performing two separate statistical tests and then comparing the outcomes (a <> b), rather than the correct approach which is to do a single test for the difference between groups (a - b <> 0).

by cl42 12y ago

return0 12y ago

Amen for #2! It's like the entire field of biology is an endless quest for low p-values, regardless if the hypothesis is even interesting. It's like people are not interested to think, they just want to publish something significantly differnt.

cl42 12y ago
haha, Sociology as well -- especially now that the web provides huge amounts of behavioral data. I much prefer how machine learning folks tend to approach predictive accuracy, though I guess that's not quite the same as understanding relationships between specific variables while controlling for others.
- sdenton4 12y ago
  There is a notion of 'feature importance,' which especially comes up in decision trees and random forests, giving a notion of how much a particular feature contributes to the overall prediction. It seems like combining predictive power with feature importance would be an interesting alternate route to demonstrating important correlations. (For example, maybe a model predicts lung cancer with 90% precision, and 'is_smoker' has a 80% feature importance.) Of course, these importances depend a lot on the other features used by the model! If you include a lot of junk features and/or exclude other important features, the importance of your pet feature will shoot up.
  - cl42 12y ago
    Hmm, interesting -- I never considered the idea of including junk features to bias model's preferences of whatever theoretically ambiguous idea you're trying to promote. That's actually brilliant.
    - mbq 12y ago
      Shameless plug; I'm a co-author of a method that leverages adding artificial junk features and removing original ones that are likely nonsense to approximate the set of all features that are relevant to the problem (rather than standard make best model, which may be pretty deceiving). https://m2.icm.edu.pl/boruta https://m2.icm.edu.pl/boruta
shasta 12y ago
Shoes reduce cholesterol.
- cl42 12y ago
  And aren't ice cream sales dependent on murder rates?
  - copperx 12y ago
    Did you just discredit Freakonomics?
Sharlin 12y ago
> they just want to publish something What's going on in the academia is a classical example of perverse incentives. If your career depends on churning out soul-crushingly bad or insignificant papers that fulfill some superficial criteria of "scienceness" then that's what you'll do. It's dangerously close to cargo cult science.

conistonwater 12y ago

> are irrationally attached to P values This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not. I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome. It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.

wodenokoto 12y ago
> so they insist on following some strict rules, whether they make sense or not. I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data. As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005
- RA_Fisher 12y ago
  Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
  - cgearhart 12y ago
    > The reason is that the p-value essentially tells you how repeatable your analysis will be Your statement contradicts the introductory paragraph of the article: > "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..." The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.

Scoundreller 12y ago

One of my favourite go-to phrases in health care is: "Statistically significant, but a clinically insignificant difference."

cl42 12y ago
I love it! That's actually a really good way of looking at it, right? Unfortunately for researchers, I'd argue that "Statistically significant, but a clinically insignificant difference." is equivalent to "Theoretically useless, and technically irrelevant." :)
6502nerdface 12y ago
In quant finance, my colleagues and I say, "OK, it's statistically significant, but is it economically significant?" I love that we have parallel sayings in totally different fields.
remarkEon 12y ago
Right, you're describing statistical significance as it relates to practical significance.

joshdick 12y ago

Addressing that first point is "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant" by Andrew Gelman and Hal Stern. PDF link: http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf http://www.stat.columbia.edu/~gelman/research/published/sign...

stdbrouw 12y ago
Nitpick: while that paper does mention the basic idea that there's no significant difference between 0.051 and 0.049, it's more about another common mistake: testing the difference between two groups by performing two separate statistical tests and then comparing the outcomes (a <> b), rather than the correct approach which is to do a single test for the difference between groups (a - b <> 0).