3 ms·

Hack Your Way to Scientific Glory (2016)

75 points by wjb3 2y ago

user_7832 2y ago
For anyone wondering about the title, it's about p-hacking - picking and choosing variables to suit your objectives. Highly recommend trying the mini-game, it's fun!
YossarianFrPrez 2y ago
They say a photo is worth a thousand words, and here, this demo is as convincing as a lot of the spilt ink regarding the replication crisis and social psychology. As someone who pursuing a PhD in the social sciences... This could be a useful tool for seniors majoring in the social sciences / who are taking stats classes.
paulpauper 2y ago
Sigh. P hacking has long been a meme. Nicholas Cage and shark attacks for example. P values are useful though if the results seem at least plausible or can be explained by some underlying process. Like a low p value between calorie restriction and weight loss.
- caddemon 2y ago
  Shark attacks isn't really p-hacking though, that's about misinterpreting/misrepresenting a real correlation. P-hacking is finding a fake correlation due to chance because you tested a lot of different things in a smallish dataset, and then reporting that correlation as meaningful without mentioning your negative results or properly explaining your entire process.
  - kjkjadksj 2y ago
    This is what I don’t fully grasp about p hacking myself. Say you test variable abc and so on all the way to z. Variable z were found to be significant. Now imagine you had the same data set and just happened to test Z and stopped there, finding it significant. According to the p hacking concept the former would be potentially unethical while the latter would be great science, however nothing about the underlying data or its association between variables has changed.
tbenst 2y ago
I know the exercise was to p-hack, but instead I decided to one-shot my attempt at the most reasonable model from first principals: - given that we are looking at a national scale, use only national politicians - use the components from Macroeconomics 101: exclude inflation as that’s on the Fed, exclude stocks as too conflated with FX and international investing alternatives - don’t needlessly withhold data Tried one hypothesis, so p-value of 0.04 is accurate. Still OK to explore if you Bonferroni correct the p-Val afterwards
wjb3 2y ago
An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies. How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?
- jarsdel 2y ago
  Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
  - nextos 2y ago
    eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration. I've found their reviews much more detailed and technical than Nature Medicine or Cell.
    - wjb3 2y ago
      When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?
      - nextos 2y ago
        eLife publishes reviews very prominently within articles, and they ask all reviewers to discuss their assessment after they have evaluated the article. I think this favors deep reviews. In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion. I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.

75 points by wjb3 2y ago

For anyone wondering about the title, it's about p-hacking - picking and choosing variables to suit your objectives. Highly recommend trying the mini-game, it's fun!

YossarianFrPrez 2y ago

They say a photo is worth a thousand words, and here, this demo is as convincing as a lot of the spilt ink regarding the replication crisis and social psychology. As someone who pursuing a PhD in the social sciences... This could be a useful tool for seniors majoring in the social sciences / who are taking stats classes.

paulpauper 2y ago

Sigh. P hacking has long been a meme. Nicholas Cage and shark attacks for example. P values are useful though if the results seem at least plausible or can be explained by some underlying process. Like a low p value between calorie restriction and weight loss.

caddemon 2y ago
Shark attacks isn't really p-hacking though, that's about misinterpreting/misrepresenting a real correlation. P-hacking is finding a fake correlation due to chance because you tested a lot of different things in a smallish dataset, and then reporting that correlation as meaningful without mentioning your negative results or properly explaining your entire process.
- kjkjadksj 2y ago
  This is what I don’t fully grasp about p hacking myself. Say you test variable abc and so on all the way to z. Variable z were found to be significant. Now imagine you had the same data set and just happened to test Z and stopped there, finding it significant. According to the p hacking concept the former would be potentially unethical while the latter would be great science, however nothing about the underlying data or its association between variables has changed.

tbenst 2y ago

I know the exercise was to p-hack, but instead I decided to one-shot my attempt at the most reasonable model from first principals: - given that we are looking at a national scale, use only national politicians - use the components from Macroeconomics 101: exclude inflation as that’s on the Fed, exclude stocks as too conflated with FX and international investing alternatives - don’t needlessly withhold data Tried one hypothesis, so p-value of 0.04 is accurate. Still OK to explore if you Bonferroni correct the p-Val afterwards

wjb3 2y ago

An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies. How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?

jarsdel 2y ago
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
- nextos 2y ago
  eLife is very rigorous, but also very selective. Most submissions don't pass editorial consideration. I've found their reviews much more detailed and technical than Nature Medicine or Cell.
  - wjb3 2y ago
    When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?
    - nextos 2y ago
      eLife publishes reviews very prominently within articles, and they ask all reviewers to discuss their assessment after they have evaluated the article. I think this favors deep reviews. In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion. I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.