4 ms·

ELI5... what’s a p-hack?

by nvahalik 8y ago

laurieg 8y ago
Doing the same experiment again and again until you get the answer you want.
- drdrey 8y ago
  In this case though I think they mean stopping the experiment early if there seems to be a positive result rather than experimenting again
SquareWheel 8y ago
Not an easy ELI5, but consider these resources. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging https://www.youtube.com/watch?v=bf3egy7TQ2Q https://www.youtube.com/watch?v=bf3egy7TQ2Q
danielvf 8y ago
“the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“ https://en.m.wikipedia.org/wiki/Data_dredging https://en.m.wikipedia.org/wiki/Data_dredging And here’s a great example of real life P-Hacking to get a catchy article about the health benefits of chocolate: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
- vitus 8y ago
  FiveThirtyEight actually has a pretty good demo of p-hacking that demonstrates how one underlying dataset can be used to derive any desired conclusion(s) by deciding which factors to include / exclude. https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/
- cryptonector 8y ago
  Torturing the data until it sings.
delecti 8y ago
In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p<0.05. A good example is this xkcd [2]. In the example, green jelly beans aren't actually related to acne, but just by chance the result seems to indicate they are. [1] https://en.m.wikipedia.org/wiki/P-value https://en.m.wikipedia.org/wiki/P-value [2] https://xkcd.com/882/ https://xkcd.com/882/
patmcc 8y ago
https://xkcd.com/882/ https://xkcd.com/882/
- vecter 8y ago
  This is not p-hacking.
  - soberhoff 8y ago
    It is.
cepth 8y ago
If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated with cancer”, a 0.05 p-value would meet the threshold to reject that null hypothesis. Generally, a lower p-value means a more statistically significant result. The problem is that 0.05 seems to be much too high of a p-value. I.e. clever experimental design and cherry picking can generate many results that are statistically significant at that level. Many academics advocate for moving to a 0.01 or even 0.001 significance threshold. Recently, in some academic fields, there’s been widespread concern that many research studies were p-hacked. See for example, this paper that blew up last year in the finance community, because it suggests a significant number of finance papers, including some seminal ones, had p-hacked results: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677. The counter-argument is that for certain scientific fields, you may never be able to reach a p-value threshold of 0.001. This means the vast majority of research couldn’t be published in journals, academics wouldn’t be able to get promoted etc.
- hammock 8y ago
  This is wrong. P hacking has nothing to do with the p being too lenient.
  - Fomite 8y ago
    This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.
    - cepth 8y ago
      > This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to address the lack of reproducibility is not something that I made up. Yes, lowering the p-value threshold does not eliminate the motivations/techniques that are necessary for p-hacking, but it can make it a lot harder, and a lot less worthwhile. If you work in academia, and it takes you much longer to now cherrypick a sample to meet a much lower p-cutoff, it seems to follow that we would see less of it. This is an excerpt from a paper soon to be published in Nature: https://imai.princeton.edu/research/files/significance.pdf https://imai.princeton.edu/research/files/significance.pdf. The key quote: 'We have diverse views about how best to improve reproducibility,and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.' With regards to the comment that you could p-hack up to any strictness, I'm not sure this is correct. If you accept the proposal laid out in that Nature paper, to lower the threshold to P<0.005, or if we go even lower to P<0.001 I don't believe that you'd be able to p-hack in any practical way. Yes, you could cherry pick a tiny sample, but any peer reviewer or colleague of yours is going to ask questions about the sample.
azernik 8y ago
Even better than examples and explanations, learn by doing: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ "You’re a social scientist with a hunch: The U.S. economy is affected by whether Republicans or Democrats are in office. Try to show that a connection exists, using real data going back to 1948. For your results to be publishable in an academic journal, you’ll need to prove that they are “statistically significant” by achieving a low enough p-value."

ELI5... what’s a p-hack?

laurieg 8y ago

Doing the same experiment again and again until you get the answer you want.

drdrey 8y ago
In this case though I think they mean stopping the experiment early if there seems to be a positive result rather than experimenting again

SquareWheel 8y ago

Not an easy ELI5, but consider these resources. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging https://www.youtube.com/watch?v=bf3egy7TQ2Q https://www.youtube.com/watch?v=bf3egy7TQ2Q

danielvf 8y ago

“the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“ https://en.m.wikipedia.org/wiki/Data_dredging https://en.m.wikipedia.org/wiki/Data_dredging And here’s a great example of real life P-Hacking to get a catchy article about the health benefits of chocolate: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...

vitus 8y ago
FiveThirtyEight actually has a pretty good demo of p-hacking that demonstrates how one underlying dataset can be used to derive any desired conclusion(s) by deciding which factors to include / exclude. https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/
cryptonector 8y ago
Torturing the data until it sings.

delecti 8y ago

In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p<0.05. A good example is this xkcd [2]. In the example, green jelly beans aren't actually related to acne, but just by chance the result seems to indicate they are. [1] https://en.m.wikipedia.org/wiki/P-value https://en.m.wikipedia.org/wiki/P-value [2] https://xkcd.com/882/ https://xkcd.com/882/

patmcc 8y ago

https://xkcd.com/882/ https://xkcd.com/882/

vecter 8y ago
This is not p-hacking.
- soberhoff 8y ago
  It is.

cepth 8y ago

If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated with cancer”, a 0.05 p-value would meet the threshold to reject that null hypothesis. Generally, a lower p-value means a more statistically significant result. The problem is that 0.05 seems to be much too high of a p-value. I.e. clever experimental design and cherry picking can generate many results that are statistically significant at that level. Many academics advocate for moving to a 0.01 or even 0.001 significance threshold. Recently, in some academic fields, there’s been widespread concern that many research studies were p-hacked. See for example, this paper that blew up last year in the finance community, because it suggests a significant number of finance papers, including some seminal ones, had p-hacked results: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677. The counter-argument is that for certain scientific fields, you may never be able to reach a p-value threshold of 0.001. This means the vast majority of research couldn’t be published in journals, academics wouldn’t be able to get promoted etc.

hammock 8y ago
This is wrong. P hacking has nothing to do with the p being too lenient.
- Fomite 8y ago
  This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.
  - cepth 8y ago
    > This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to address the lack of reproducibility is not something that I made up. Yes, lowering the p-value threshold does not eliminate the motivations/techniques that are necessary for p-hacking, but it can make it a lot harder, and a lot less worthwhile. If you work in academia, and it takes you much longer to now cherrypick a sample to meet a much lower p-cutoff, it seems to follow that we would see less of it. This is an excerpt from a paper soon to be published in Nature: https://imai.princeton.edu/research/files/significance.pdf https://imai.princeton.edu/research/files/significance.pdf. The key quote: 'We have diverse views about how best to improve reproducibility,and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.' With regards to the comment that you could p-hack up to any strictness, I'm not sure this is correct. If you accept the proposal laid out in that Nature paper, to lower the threshold to P<0.005, or if we go even lower to P<0.001 I don't believe that you'd be able to p-hack in any practical way. Yes, you could cherry pick a tiny sample, but any peer reviewer or colleague of yours is going to ask questions about the sample.

azernik 8y ago

Even better than examples and explanations, learn by doing: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ "You’re a social scientist with a hunch: The U.S. economy is affected by whether Republicans or Democrats are in office. Try to show that a connection exists, using real data going back to 1948. For your results to be publishable in an academic journal, you’ll need to prove that they are “statistically significant” by achieving a low enough p-value."