6 ms·

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we re

The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.

buboard 7y ago
Scientists have a duty to communicate what is true and what is not though. Passing that judgement duty to the general public is irresponsible and unwise (it's easy to claim anything with flawed / misleading statistics). Ideally someone would have come up with a framework that is better than statistic tests to justify levels of truth, but so far we dont have one
- mkl 7y ago
  When you've got empirically gathered data, pretty much every type of analysis you can do with it comes under the umbrella of "statistics". I think any better truth measuring framework can only be statistical.
  - buboard 7y ago
    There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.
    - bigfudge 7y ago
      They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.
      - mkl 7y ago
        They pretty much agree that statistics is the tool to use, but disagree on the (important) details.
      - deleted 7y ago
        [deleted]
      - buboard 7y ago
        Scientists (should) never really "agree", they just concede temporarily preparing their next attack. statistic tests are the de facto standard that everyone concedes to, and has not been replaced.
- merpnderp 7y ago
  Richard Feynman said anything you can't prove yourself must be taken on pure faith and most of the public must take everything on pure faith as somehow they've left school without the faintest idea how basic stats work.
  - buboard 7y ago
    Feynman was talking to scientists. I do think that science has to provide answers, not just data. The public interprets every datum in a myriad of ways, and does not have a mechanism to establish consensus, hence you would end up with constant crisis.
    - marknutter 7y ago
      Who says we're not in a constant crisis?
CreRecombinase 7y ago
This is a a really common misconception about p values (that they can be interpreted as p(H0|x), or "probability of the null hypothesis given the data") when a p-value is in fact p(x|H0), or "probability of observing data at least this extreme given that the null hypothesis is true
- TallGuyShort 7y ago
  Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.
  - SquishyPanda23 7y ago
    The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.
    - CrazyStat 7y ago
      With some caveats: the value you get for P(H|x) might be very sensitive to the priors you choose, and most people are not thoughtful enough about this.
      - SquishyPanda23 7y ago
        Yes that is true, but there is a sense in which that just makes explicit a problem that already exists.
    - pfortuny 7y ago
      Wait wait wait: you need the priors, also. Where do they come from?
      - 7y ago
krcz 7y ago
It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".
- onlyrealcuzzo 7y ago
  It seems like an education fail to me. Most people don't know the very basics of stats and we live in a world that's highly probabilistic. It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
  - TallGuyShort 7y ago
    This guy is one of my favorite TED speakers, and he makes a strong argument for exactly this: https://www.ted.com/talks/arthur_benjamin_s_formula_for_changing_math_education https://www.ted.com/talks/arthur_benjamin_s_formula_for_chan....
    - susiecambria 7y ago
      Thanks for sharing! (writes the girl who barely passed pre-calc)
  - grawprog 7y ago
    I never learned statistics until my second year in university, it was really mind opening, especially with showing me just how much poorly done statistics with tiny sample sizes or misrepresented data are quoted and used as facts. I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.
  - 7y ago
ludamad 7y ago
"3% probability we're just seeing a pattern by accident" in fact still sounds a lot better than it is if there is a positive result bias, as you'll miss the "but 99 percent of scientists who attempted this did not see a pattern"
roenxi 7y ago
We use hard cutoffs for a bunch of things, they aren't perfect but they are fine. The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English. It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter. This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean. Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance. For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.
- kerkeslager 7y ago
  This is a really good point. In fact, failures to reproduce "landmark" experiments have low significance (statistics word) because they find a result that is highly probable if the null hypothesis is true. But failures to reproduce have high significance (colloquial English word) because they indicate that widely-held beliefs are wrong. So in some of the most important cases, the statistics and colloquial English meanings of the words are exactly the opposite. I don't know if there's a good solution to this, though--it's hard to change organically-emergent trends in language even if we did come up with a better term.
- abakker 7y ago
  This is an excellent point. In my work, the interpretation error I see the most among people around be is to confuse significance with the magnitude of the predicted effect. I.e. if you use OLS to create a model, it is possible to have a statistically significant beta, with essentially no magnitude. If you look at research on carcinogens, it is frequent to see something like "it is statistically significant that eating chemical X increases your risk of cancer" but when you read on, the magnitude of the effect is "increases risk by .0023%, for a cancer that already has a .6% incidence in the population". My point is that people who are not statisticians (or people who use statistics), tend to be concerned with the magnitude of an effect, and frequently misconstrue the statistical significance for that magnitude.
platz 7y ago
At the end of the day, people need to make a decision about something, and I suspect this is what is forcing the importance of this binary distinction. My decision to select, implement, go, or no go cannot itself be a probability distribution.
- TallGuyShort 7y ago
  Decision to select shouldn't be the subject of academic or scientific papers in the first place. There's far more context required in decision making than just how strong a correlation is.
  - platz 7y ago
    I agree, but you might see how this error or preference creeps in psychologically
btilly 7y ago
I disbelieve. The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs. Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly because it is so easily misunderstood. You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood. Worse yet, p-values are sensitive to particulars of experimental design that logically should never matter to inferences. My aunt and uncle provide a classic example. They wanted a son and a daughter. They had 6 sons then a daughter. Are they biased towards one gender? The p-value for this question is .03125 (we'd see this much evidence against with all boys, all girls, last girl, last boy, so 4/2^7). But if I instead said that they planned to have 7 children, and had 6 sons then a daughter, the p-value changes to 0.125 because the one child out could have been anywhere in succession. But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences. (This example is not made up. When Lorna found that they had a daughter, she told Bill to get a vasectomy from the delivery table. By another crazy coincidence, their 7 children were also born on all the days of the week.)
- RosanaAnaDana 7y ago
  Well, sort-of. Its not just about p-values or the likelyhood of making a particular type of error by agreeing with a thesis. There is a fundamental misunderstanding that most people, and even many scientists make with regards to the philosophical interpretation of 'truth' using the scientific method. By definition, scientific 'truth' is always fallible, and the vast majority of people have a very hard time dealing with this concept. A better explanation or additional data, or better data is always possible and needs to be possible for science to not be dogma. This is fundamentally different than how people interpret truth and understand what the word means. All scientific truths are replaceable.
- ghkbrew 7y ago
  > But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments: 1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)? 2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?
WhompingWindows 7y ago
P-values are much more problematic at the alpha=.05 threshold than the alpha=.005 or alpha=.01 thresholds. In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are <.0001, meaning there is SO much data that even a very very small effect size can be found "significant." The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?
- mswen 7y ago
  Yes, thank you. Practical significance. You cannot get away with just running statistical test or deep learning models. You often need to ask whether a change in product, service or process to take advantage of some trend or difference between customers segment will actually be worth doing. That involves getting your hands dirty in the business, trying to identify all the costs and side-effects of a change, and then how much of a size effect you will actually capture once the changes are made. My views in more detail [1] [1] http://computationalimagination.com/article_practical_significance.php http://computationalimagination.com/article_practical_signif...
rjf72 7y ago
I agree with you, but I think there is an even more fundamental issue. P-values really don't tell you anything. They can refute a bad hypothesis but they cannot validate a good one. This [1] is such an awesome site. It lists a large number of spurious correlations. It's quite remarkable how strong many of them are. For instance there is a 99.79% correlation between suicides by hanging and US spending on science/space/technology. The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship. 538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal. [1] - http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations [2] - https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/
usgroup 7y ago
I’m a Bayesian flavoured, and the “problem” IMO is that model driven statistics are really hard and beyond most scientists (people generally tbh). If we required scientists to be good statisticians there’d be far fewer scientists.
- godelski 7y ago
  Or we'd have just as many, but better at statistics. P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful. Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.
analog31 7y ago
I have a rule of thumb: Multiply p by 10. In other words, p = 0.05 is as good as a coin toss. This is consistent with the so called replication crisis. In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects. I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.
tschwimmer 7y ago
I think this is part of the problem, but the more fundamental issue is that people are extremely bad about probabilistic reasoning. Even if we said "3% probability we're just seeing a pattern by accident" most people unconsciously round this down to zero. People that work with computers often are slightly better about reasoning about probabilities because they often deal with high frequency events. A 1% error rate across 10 million events a day leads to 100,000 errors. If those events are transactions, payments, file uploads etc, a 1% chance starts to look very common. But even computer people are not immune to incorrectly reasoning about probabilities. I suspect there is some fundamental limitation at the cognitive level.
jredwards 7y ago
While I agree with that completely, the coinciding problem that I've run into time and time again is the temptation to accept statistical significance but ignore the size of the effect. You might see p=0.00001 but if the size of the effect irrelevant, then the statistical significance of the relationship is still not something I care about. Great, orange juice shrinks tumors by 0.000001%. I'm still going with chemo, thanks.

buboard 7y ago

Scientists have a duty to communicate what is true and what is not though. Passing that judgement duty to the general public is irresponsible and unwise (it's easy to claim anything with flawed / misleading statistics). Ideally someone would have come up with a framework that is better than statistic tests to justify levels of truth, but so far we dont have one

mkl 7y ago
When you've got empirically gathered data, pretty much every type of analysis you can do with it comes under the umbrella of "statistics". I think any better truth measuring framework can only be statistical.
- buboard 7y ago
  There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.
  - bigfudge 7y ago
    They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.
    - mkl 7y ago
      They pretty much agree that statistics is the tool to use, but disagree on the (important) details.
    - deleted 7y ago
      [deleted]
    - buboard 7y ago
      Scientists (should) never really "agree", they just concede temporarily preparing their next attack. statistic tests are the de facto standard that everyone concedes to, and has not been replaced.
merpnderp 7y ago
Richard Feynman said anything you can't prove yourself must be taken on pure faith and most of the public must take everything on pure faith as somehow they've left school without the faintest idea how basic stats work.
- buboard 7y ago
  Feynman was talking to scientists. I do think that science has to provide answers, not just data. The public interprets every datum in a myriad of ways, and does not have a mechanism to establish consensus, hence you would end up with constant crisis.
  - marknutter 7y ago
    Who says we're not in a constant crisis?

CreRecombinase 7y ago

This is a a really common misconception about p values (that they can be interpreted as p(H0|x), or "probability of the null hypothesis given the data") when a p-value is in fact p(x|H0), or "probability of observing data at least this extreme given that the null hypothesis is true

TallGuyShort 7y ago
Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.
- SquishyPanda23 7y ago
  The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute. Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way. You can actually compute p(H|x) if you use Bayesian stats.
  - CrazyStat 7y ago
    With some caveats: the value you get for P(H|x) might be very sensitive to the priors you choose, and most people are not thoughtful enough about this.
    - SquishyPanda23 7y ago
      Yes that is true, but there is a sense in which that just makes explicit a problem that already exists.
  - pfortuny 7y ago
    Wait wait wait: you need the priors, also. Where do they come from?
    - 7y ago

krcz 7y ago

It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".

onlyrealcuzzo 7y ago
It seems like an education fail to me. Most people don't know the very basics of stats and we live in a world that's highly probabilistic. It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
- TallGuyShort 7y ago
  This guy is one of my favorite TED speakers, and he makes a strong argument for exactly this: https://www.ted.com/talks/arthur_benjamin_s_formula_for_changing_math_education https://www.ted.com/talks/arthur_benjamin_s_formula_for_chan....
  - susiecambria 7y ago
    Thanks for sharing! (writes the girl who barely passed pre-calc)
- grawprog 7y ago
  I never learned statistics until my second year in university, it was really mind opening, especially with showing me just how much poorly done statistics with tiny sample sizes or misrepresented data are quoted and used as facts. I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.
- 7y ago

ludamad 7y ago

"3% probability we're just seeing a pattern by accident" in fact still sounds a lot better than it is if there is a positive result bias, as you'll miss the "but 99 percent of scientists who attempted this did not see a pattern"

roenxi 7y ago

We use hard cutoffs for a bunch of things, they aren't perfect but they are fine. The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English. It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter. This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean. Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance. For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.

kerkeslager 7y ago
This is a really good point. In fact, failures to reproduce "landmark" experiments have low significance (statistics word) because they find a result that is highly probable if the null hypothesis is true. But failures to reproduce have high significance (colloquial English word) because they indicate that widely-held beliefs are wrong. So in some of the most important cases, the statistics and colloquial English meanings of the words are exactly the opposite. I don't know if there's a good solution to this, though--it's hard to change organically-emergent trends in language even if we did come up with a better term.
abakker 7y ago
This is an excellent point. In my work, the interpretation error I see the most among people around be is to confuse significance with the magnitude of the predicted effect. I.e. if you use OLS to create a model, it is possible to have a statistically significant beta, with essentially no magnitude. If you look at research on carcinogens, it is frequent to see something like "it is statistically significant that eating chemical X increases your risk of cancer" but when you read on, the magnitude of the effect is "increases risk by .0023%, for a cancer that already has a .6% incidence in the population". My point is that people who are not statisticians (or people who use statistics), tend to be concerned with the magnitude of an effect, and frequently misconstrue the statistical significance for that magnitude.

platz 7y ago

At the end of the day, people need to make a decision about something, and I suspect this is what is forcing the importance of this binary distinction. My decision to select, implement, go, or no go cannot itself be a probability distribution.

TallGuyShort 7y ago
Decision to select shouldn't be the subject of academic or scientific papers in the first place. There's far more context required in decision making than just how strong a correlation is.
- platz 7y ago
  I agree, but you might see how this error or preference creeps in psychologically

btilly 7y ago

I disbelieve. The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs. Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly because it is so easily misunderstood. You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood. Worse yet, p-values are sensitive to particulars of experimental design that logically should never matter to inferences. My aunt and uncle provide a classic example. They wanted a son and a daughter. They had 6 sons then a daughter. Are they biased towards one gender? The p-value for this question is .03125 (we'd see this much evidence against with all boys, all girls, last girl, last boy, so 4/2^7). But if I instead said that they planned to have 7 children, and had 6 sons then a daughter, the p-value changes to 0.125 because the one child out could have been anywhere in succession. But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences. (This example is not made up. When Lorna found that they had a daughter, she told Bill to get a vasectomy from the delivery table. By another crazy coincidence, their 7 children were also born on all the days of the week.)

RosanaAnaDana 7y ago
Well, sort-of. Its not just about p-values or the likelyhood of making a particular type of error by agreeing with a thesis. There is a fundamental misunderstanding that most people, and even many scientists make with regards to the philosophical interpretation of 'truth' using the scientific method. By definition, scientific 'truth' is always fallible, and the vast majority of people have a very hard time dealing with this concept. A better explanation or additional data, or better data is always possible and needs to be possible for science to not be dogma. This is fundamentally different than how people interpret truth and understand what the word means. All scientific truths are replaceable.
ghkbrew 7y ago
> But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments: 1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)? 2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?

WhompingWindows 7y ago

P-values are much more problematic at the alpha=.05 threshold than the alpha=.005 or alpha=.01 thresholds. In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are <.0001, meaning there is SO much data that even a very very small effect size can be found "significant." The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?

mswen 7y ago
Yes, thank you. Practical significance. You cannot get away with just running statistical test or deep learning models. You often need to ask whether a change in product, service or process to take advantage of some trend or difference between customers segment will actually be worth doing. That involves getting your hands dirty in the business, trying to identify all the costs and side-effects of a change, and then how much of a size effect you will actually capture once the changes are made. My views in more detail [1] [1] http://computationalimagination.com/article_practical_significance.php http://computationalimagination.com/article_practical_signif...

rjf72 7y ago

I agree with you, but I think there is an even more fundamental issue. P-values really don't tell you anything. They can refute a bad hypothesis but they cannot validate a good one. This [1] is such an awesome site. It lists a large number of spurious correlations. It's quite remarkable how strong many of them are. For instance there is a 99.79% correlation between suicides by hanging and US spending on science/space/technology. The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship. 538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal. [1] - http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations [2] - https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/

usgroup 7y ago

I’m a Bayesian flavoured, and the “problem” IMO is that model driven statistics are really hard and beyond most scientists (people generally tbh). If we required scientists to be good statisticians there’d be far fewer scientists.

godelski 7y ago
Or we'd have just as many, but better at statistics. P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful. Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.

analog31 7y ago

I have a rule of thumb: Multiply p by 10. In other words, p = 0.05 is as good as a coin toss. This is consistent with the so called replication crisis. In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects. I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.

tschwimmer 7y ago

I think this is part of the problem, but the more fundamental issue is that people are extremely bad about probabilistic reasoning. Even if we said "3% probability we're just seeing a pattern by accident" most people unconsciously round this down to zero. People that work with computers often are slightly better about reasoning about probabilities because they often deal with high frequency events. A 1% error rate across 10 million events a day leads to 100,000 errors. If those events are transactions, payments, file uploads etc, a 1% chance starts to look very common. But even computer people are not immune to incorrectly reasoning about probabilities. I suspect there is some fundamental limitation at the cognitive level.

jredwards 7y ago

While I agree with that completely, the coinciding problem that I've run into time and time again is the temptation to accept statistical significance but ignore the size of the effect. You might see p=0.00001 but if the size of the effect irrelevant, then the statistical significance of the relationship is still not something I care about. Great, orange juice shrinks tumors by 0.000001%. I'm still going with chemo, thanks.