8 ms·

The incentives of academia naturally select for less reliable results (2016)

106 points by dom0 9y ago

nonbel 9y ago
The proposed "solutions" of pre-registration and increase sample size are not going to work because they are not fixing the underlying problem of NHST use (where a researcher tests a nil null hypothesis of zero effect and then concludes their favorite explanation is correct if they reject this null hypothesis). The solution is to go back to the old way of science (pre-1940): 1) Collect data and figure out what is repeatable and consistent (eg come up with "natural laws"). 2) Come up with theories that explain the observations from #1 that also make precise predictions about some different type of data. Reward researchers who figure out methods that lead to reliable, repeatable observations and those that make precise predictions (if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague).
- madhadron 9y ago
  Your solution is historically inaccurate. Statistical tests of significance were created as a tool to improve the objectivity of evaluating results. Garbage in, garbage out still applies. Nor is it relevant. Statistical techniques are a symptom, not an underlying cause. Indeed, everything in the article or in discussions of this that I see is always treated symptoms far from the causes. Let's do a quick five whys: Why are publications not reproducible? Because positive results are being published as fast as possible with little checking on them. Why are they publishing so? Because the people involved need publications to get out of their current transient position and secure another transient position, with hopes of someday being able to settle down somewhere. Why do they need to secure their next transient position? Because they have been inculcated that this is the career to be in by their milieu and that is the only way to stay in it. Why is that the only way to stay in it? Because there are no permanent positions that aren't at the very end of this gauntlet. Why are there no permanent positions except at the end of the gauntlet? Because this is a pyramid scheme built on an apprenticeship system. Don't get distracted by all the other stuff. It's the fundamental structure of academic science that leads to the rest of this.
  - nonbel 9y ago
    So let's go to the beginning: >"Because positive results are being published as fast as possible with little checking on them." Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
    - IanCal 9y ago
      Replicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
      - dgfgfdagasdfgfa 9y ago
        Why, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result. Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
        IanCal 9y ago
        They are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work. Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
        nonbel 9y ago
- IanCal 9y ago
  > if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague Can nobody publish these results? Is it not a sensible building block on the way to precise predictions?
  - avs733 9y ago
    context matters...in both the research and way it is reported. correlation is not too vague, but implying causality from correlation without further tests is conversely... assuming that a paper that only reports correlation or reports correlation as well as further evidence of causality (i.e., prior work on causal mechanisms, qualitative evidence, etc.) is faulty simply because it has correlation is equally problematic. The major problem of the article is that it displays the same flaws as many of the underlying problems in science itself: It makes massive generalizations without nuance or appreciation of context. The problems are paradigmatic (i.e., Kuhnian) and the fact that they appear in the research AND in the critiques of the research are far more problematic than a p of .05.
    - dgfgfdagasdfgfa 9y ago
      I think "cultural" is a better term than "paradigmatic"; the latter typically refers to ontological revolutions, not process revolutions.
      - avs733 9y ago
        Fair point of clarification...I was specifically referring to the ontological and epistemological components of the argument. The perception of many scientists (especially young ones) that their research is inherently independent of them as a researcher is a big problem. These are issues that do not appear at surface or even values levels...they are base level assumptions about knowledge that manifest very subtly.
  - deleted 9y ago
    [deleted]
  - nonbel 9y ago
avs733 9y ago
>Low statistical power is an obvious symptom of weak research. No it isn't. It is a simple one but it isn't obvious. Low statistical power is just one of many indicators of research quality. Qualitative research with sample sizes in the single digits can be done well and research with large statistical power can be done poorly. The measure of quality in research is much more conceptual and cannot be reduced to rule based measures without failing completely.
- marcosdumay 9y ago
  It shouldn't even be an indicator. The less certain you are about your hypothesis set, the less statistical power you should seek on your research. That way you can optimally filter the set and reject most falsehoods faster and cheaper. But it is currently a useful indicator. This is a symptom of a problem by itself.
- tlb 9y ago
  Statistical power includes effect size, so "qualitative" research (with large, easily measurable effects) can have high statistical power with only a few samples. https://en.wikipedia.org/wiki/Statistical_power https://en.wikipedia.org/wiki/Statistical_power
  - avs733 9y ago
    I'm really curious as to why you put qualitative in quotes.
deleted 9y ago
[deleted]

106 points by dom0 9y ago

The proposed "solutions" of pre-registration and increase sample size are not going to work because they are not fixing the underlying problem of NHST use (where a researcher tests a nil null hypothesis of zero effect and then concludes their favorite explanation is correct if they reject this null hypothesis). The solution is to go back to the old way of science (pre-1940): 1) Collect data and figure out what is repeatable and consistent (eg come up with "natural laws"). 2) Come up with theories that explain the observations from #1 that also make precise predictions about some different type of data. Reward researchers who figure out methods that lead to reliable, repeatable observations and those that make precise predictions (if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague).

madhadron 9y ago
Your solution is historically inaccurate. Statistical tests of significance were created as a tool to improve the objectivity of evaluating results. Garbage in, garbage out still applies. Nor is it relevant. Statistical techniques are a symptom, not an underlying cause. Indeed, everything in the article or in discussions of this that I see is always treated symptoms far from the causes. Let's do a quick five whys: Why are publications not reproducible? Because positive results are being published as fast as possible with little checking on them. Why are they publishing so? Because the people involved need publications to get out of their current transient position and secure another transient position, with hopes of someday being able to settle down somewhere. Why do they need to secure their next transient position? Because they have been inculcated that this is the career to be in by their milieu and that is the only way to stay in it. Why is that the only way to stay in it? Because there are no permanent positions that aren't at the very end of this gauntlet. Why are there no permanent positions except at the end of the gauntlet? Because this is a pyramid scheme built on an apprenticeship system. Don't get distracted by all the other stuff. It's the fundamental structure of academic science that leads to the rest of this.
- nonbel 9y ago
  So let's go to the beginning: >"Because positive results are being published as fast as possible with little checking on them." Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
  - IanCal 9y ago
    Replicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
    - dgfgfdagasdfgfa 9y ago
      Why, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result. Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
      - IanCal 9y ago
        They are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work. Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
        nonbel 9y ago
IanCal 9y ago
> if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague Can nobody publish these results? Is it not a sensible building block on the way to precise predictions?
- avs733 9y ago
  context matters...in both the research and way it is reported. correlation is not too vague, but implying causality from correlation without further tests is conversely... assuming that a paper that only reports correlation or reports correlation as well as further evidence of causality (i.e., prior work on causal mechanisms, qualitative evidence, etc.) is faulty simply because it has correlation is equally problematic. The major problem of the article is that it displays the same flaws as many of the underlying problems in science itself: It makes massive generalizations without nuance or appreciation of context. The problems are paradigmatic (i.e., Kuhnian) and the fact that they appear in the research AND in the critiques of the research are far more problematic than a p of .05.
  - dgfgfdagasdfgfa 9y ago
    I think "cultural" is a better term than "paradigmatic"; the latter typically refers to ontological revolutions, not process revolutions.
    - avs733 9y ago
      Fair point of clarification...I was specifically referring to the ontological and epistemological components of the argument. The perception of many scientists (especially young ones) that their research is inherently independent of them as a researcher is a big problem. These are issues that do not appear at surface or even values levels...they are base level assumptions about knowledge that manifest very subtly.
- deleted 9y ago
  [deleted]
- nonbel 9y ago

avs733 9y ago

>Low statistical power is an obvious symptom of weak research. No it isn't. It is a simple one but it isn't obvious. Low statistical power is just one of many indicators of research quality. Qualitative research with sample sizes in the single digits can be done well and research with large statistical power can be done poorly. The measure of quality in research is much more conceptual and cannot be reduced to rule based measures without failing completely.

marcosdumay 9y ago
It shouldn't even be an indicator. The less certain you are about your hypothesis set, the less statistical power you should seek on your research. That way you can optimally filter the set and reject most falsehoods faster and cheaper. But it is currently a useful indicator. This is a symptom of a problem by itself.
tlb 9y ago
Statistical power includes effect size, so "qualitative" research (with large, easily measurable effects) can have high statistical power with only a few samples. https://en.wikipedia.org/wiki/Statistical_power https://en.wikipedia.org/wiki/Statistical_power
- avs733 9y ago
  I'm really curious as to why you put qualitative in quotes.

deleted 9y ago

[deleted]