3 ms·

You set out to test an idea. Our statistical toolkit can’t prove the idea so we say “if I check out this idea and I think the results could only have happened b

You set out to test an idea. Our statistical toolkit can’t prove the idea so we say “if I check out this idea and I think the results could only have happened by in my expected way by random chance 5% of the time, I’ll conclude I was correct. You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring. Well duh. It would be weird if you didn’t find such an outcome. This particular study openly acknowledged doing this, and also did some creative binning of their numbers to help nudge the stats. You will always be able to find something. P hacking doesn’t mean the thing you found is false, but it is at best a sloppy estimate of something to explore further. The more egregious examples of p hacking is adjusting outliers and filtering out specific datapoints until you can make the data claim whatever you want.

groestl 3y ago
Or, in the form of a comic strip https://xkcd.com/882/ https://xkcd.com/882/ (I have this in my head when p-hacking comes up)
- jncfhnb 3y ago
  A bit. But you can test 20 ideas. This is not so bad. We have tools to scrutinize such results like the false discovery rate and related concepts. The real problem is when you set out to test 20 things, and they don’t work, so you test 20 more things; and then much worse when you only tell us about the analysis that lead to the 1/40 that mattered. The thing about p hacking specifically is changing your analysis until it finds something that looks good. A lot of people don’t understand math and will do this and not understand that they’re gaming it. A lot of people will understand this and do it anyway because academia is soul crushing and they need to publish.
CoastalCoder 3y ago
Thanks for the explanation! > You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring. I think this ^^^ is the part that's tripping me up: the notion of "registered ideas". Suppose we have 21 researchers, each testing one of those 21 ideas. They pool their funds and hire someone to generate that one data set. 20 of those researchers would find their ideas unsupported by the data. But one researcher would find their idea was supported. And that's the only study that gets published in Nature. I don't hear about this ^^^^ issue being criticized the same way P-hacking is. But AFAICT it's a similar kind of statistical factor.
- Name_Chawps 3y ago
  It is criticized frequently, and study pre-registration has become the norm already in some fields.
  - CoastalCoder 3y ago
    Thanks. Then maybe it's something else I'm having trouble with: I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test). IIUC, the criticism of P-hacking hinges on such subjective details. If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?
    - jncfhnb 3y ago
      It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. If we are testing everything we can think of, we need to be more conservative about our statistical tests to reflect that. That is, helping to determine if what you found is actually useful or not. Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem. If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin. The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
      - CoastalCoder 3y ago
        > It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?

groestl 3y ago

Or, in the form of a comic strip https://xkcd.com/882/ https://xkcd.com/882/ (I have this in my head when p-hacking comes up)

jncfhnb 3y ago
A bit. But you can test 20 ideas. This is not so bad. We have tools to scrutinize such results like the false discovery rate and related concepts. The real problem is when you set out to test 20 things, and they don’t work, so you test 20 more things; and then much worse when you only tell us about the analysis that lead to the 1/40 that mattered. The thing about p hacking specifically is changing your analysis until it finds something that looks good. A lot of people don’t understand math and will do this and not understand that they’re gaming it. A lot of people will understand this and do it anyway because academia is soul crushing and they need to publish.

CoastalCoder 3y ago

Thanks for the explanation! > You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring. I think this ^^^ is the part that's tripping me up: the notion of "registered ideas". Suppose we have 21 researchers, each testing one of those 21 ideas. They pool their funds and hire someone to generate that one data set. 20 of those researchers would find their ideas unsupported by the data. But one researcher would find their idea was supported. And that's the only study that gets published in Nature. I don't hear about this ^^^^ issue being criticized the same way P-hacking is. But AFAICT it's a similar kind of statistical factor.

Name_Chawps 3y ago
It is criticized frequently, and study pre-registration has become the norm already in some fields.
- CoastalCoder 3y ago
  Thanks. Then maybe it's something else I'm having trouble with: I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test). IIUC, the criticism of P-hacking hinges on such subjective details. If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?
  - jncfhnb 3y ago
    It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. If we are testing everything we can think of, we need to be more conservative about our statistical tests to reflect that. That is, helping to determine if what you found is actually useful or not. Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem. If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin. The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
    - CoastalCoder 3y ago
      > It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?