4 ms·

I would use Andrew Gelman and John Ioannidis as prototypes for the Good Critic and Bad Critic, respectively. Gelman does real, solid work and confines his righ

by cgiles 9y ago

I would use Andrew Gelman and John Ioannidis as prototypes for the Good Critic and Bad Critic, respectively. Gelman does real, solid work and confines his righteous takedowns to a side hobby. And his criticisms are directed at individual, specific cases, with evidence, and he reserves most of his wrath for repeat offenders rather than one-off mistakes, which are unavoidable given enough projects. He confines criticisms to his area of expertise, which is statistics. Ioannidis' claim to fame is writing a hand-waving, philosophical argument in order to cast doubt on all research at once. Followed up shortly by a paper that implied (but didn't directly state!) that regression to the mean on replication implies error or fabrication in the original. And he has ridden this sort of tired argument into a lucrative and prestigious career and Stanford appointment. Never mind the collateral damage on funding and morale. My theory is that people pay attention to Ioannidis and wring their hands about these highly generic issues for the same reason most people pretend to be highly, highly concerned about sexual harassment/discrimination in the workplace -- they're praying that if they act concerned enough, the hammer will fall somewhere else. But indiscriminate criticism like this is highly toxic to science (or anywhere else, for that matter, and has the side effect of letting real bad actors off the hook). A further moral of the story is that, for better or worse, I don't trust and generally ignore criticisms that do not come from someone who has invested time into a field and made useful contributions to it. 'Full-time critics' strike me as generally sleazy and opportunistic, and I doubt the sincerity of their desire to 'improve' a field if all they can do is tear down others' work and not provide any of their own.

curuinor 9y ago
The fact that Ioannidis stakes his claim to fame as attacks on medical research is no evidence that he is a crank: it's evidence that medical research is a target-rich environment.
- cgiles 9y ago
  No doubt it is a target-rich environment. However, his arguments are so generic that they could apply to any field studying anything multifactorial or with high variance. Fundamentally, however, I don't believe, for example, that physicists are more scrupulous/intelligent/honest than biologists, who are more so than psychologists. The irreproducibility rates in a field have to be primarily caused by the nature of the system studied in that field rather than some inherent attributes of the researchers. So any field will have some constant level of irreproducibility which which essentially cannot be changed. But some researchers do better than others. My fundamental point is that if you want to change what can be changed, and genuinely improve a field, you have to focus the criticisms on specific bad actors rather than target an entire field at once and leverage a near-tautology (more complex systems yield more irreproducible results) into a 20-year tirade/career. Also, helpfully for people like Ioannidis, a statement like "field X has lots of errors and irreproducibility" is non-falsifiable, and indeed will be true no matter what, so he is taking no risk with his criticisms -- even though they do more damage to public perception, morale, and funding, and help less to change things than targeted criticisms would do. If I sound irritated by this sort of thing, it is because my graduate advisor dabbled in "meta-science" / "error correction" and I stoutly refused to have anything to do with those projects. I thought, and still think, that it is considerably more valuable to try to build my own solid lego tower despite the difficulties rather than knocking down other people's, despite the latter being considerably easier and a fairly simple and reliable route to a high citation count. In the end, irreproducible or erroneous papers in science just get ignored rather than retracted, and I think that's fine.
AlexCoventry 9y ago
> Ioannidis' claim to fame is writing a hand-waving, philosophical argument in order to cast doubt on all research at once. It's solid probabilistic reasoning about the dominant statistical methodology, and its conclusions have been empirically demonstrated in multiple fields.
- cgiles 9y ago
  It is solid probabilistic reasoning if you accept: - A publication bias term u that is pulled out of thin air for different scenarios. - A parameter R that reflects the unknowable proportion of true relationships/hypotheses in a field compared to the universe of possible hypotheses. The pre-study odds he calculates are determined entirely by R. Actually this paper provides a good framework for determining what factors will affect the PPV of a field. In that regard it was a good contribution. But the enormous leap to the title -- which can only be achieved by massive assumptions about R and u -- was baseless fearmongering and demagoguery. If one wants to address these issues at the 50,000 feet level, it would be far better to look at something concrete, such as reproducibility rates, as indeed many have done, rather than models based on pre-study odds. If we actually knew the pre-study odds, or IOW the proportion of hypotheses in hypothesis space that are true, then we could just correct our p-values for that and be done with it. Although I suppose that many Bayesians would not see any problem with abstracting away everything we don't know into priors R and u, making wild guesses about them, and drawing conclusions. We do have a replication crisis on our hands. But the pre-study odds, whatever they are, are unchangeable. If this paper had been framed in terms of "the lower the pre-study odds, the higher the power will need to be to compensate to get an acceptable level of reproducibility", I would accept it wholeheartedly, although then it would have been a simple and obvious statement rather than a citation-grabber.
  - AlexCoventry 9y ago
    Of course pre-study odds are low in fields querying complex systems on the basis of extremely weak theoretical frameworks.
    - 9y ago

by cgiles 9y ago

curuinor 9y ago

The fact that Ioannidis stakes his claim to fame as attacks on medical research is no evidence that he is a crank: it's evidence that medical research is a target-rich environment.

cgiles 9y ago
No doubt it is a target-rich environment. However, his arguments are so generic that they could apply to any field studying anything multifactorial or with high variance. Fundamentally, however, I don't believe, for example, that physicists are more scrupulous/intelligent/honest than biologists, who are more so than psychologists. The irreproducibility rates in a field have to be primarily caused by the nature of the system studied in that field rather than some inherent attributes of the researchers. So any field will have some constant level of irreproducibility which which essentially cannot be changed. But some researchers do better than others. My fundamental point is that if you want to change what can be changed, and genuinely improve a field, you have to focus the criticisms on specific bad actors rather than target an entire field at once and leverage a near-tautology (more complex systems yield more irreproducible results) into a 20-year tirade/career. Also, helpfully for people like Ioannidis, a statement like "field X has lots of errors and irreproducibility" is non-falsifiable, and indeed will be true no matter what, so he is taking no risk with his criticisms -- even though they do more damage to public perception, morale, and funding, and help less to change things than targeted criticisms would do. If I sound irritated by this sort of thing, it is because my graduate advisor dabbled in "meta-science" / "error correction" and I stoutly refused to have anything to do with those projects. I thought, and still think, that it is considerably more valuable to try to build my own solid lego tower despite the difficulties rather than knocking down other people's, despite the latter being considerably easier and a fairly simple and reliable route to a high citation count. In the end, irreproducible or erroneous papers in science just get ignored rather than retracted, and I think that's fine.

AlexCoventry 9y ago

> Ioannidis' claim to fame is writing a hand-waving, philosophical argument in order to cast doubt on all research at once. It's solid probabilistic reasoning about the dominant statistical methodology, and its conclusions have been empirically demonstrated in multiple fields.

cgiles 9y ago
It is solid probabilistic reasoning if you accept: - A publication bias term u that is pulled out of thin air for different scenarios. - A parameter R that reflects the unknowable proportion of true relationships/hypotheses in a field compared to the universe of possible hypotheses. The pre-study odds he calculates are determined entirely by R. Actually this paper provides a good framework for determining what factors will affect the PPV of a field. In that regard it was a good contribution. But the enormous leap to the title -- which can only be achieved by massive assumptions about R and u -- was baseless fearmongering and demagoguery. If one wants to address these issues at the 50,000 feet level, it would be far better to look at something concrete, such as reproducibility rates, as indeed many have done, rather than models based on pre-study odds. If we actually knew the pre-study odds, or IOW the proportion of hypotheses in hypothesis space that are true, then we could just correct our p-values for that and be done with it. Although I suppose that many Bayesians would not see any problem with abstracting away everything we don't know into priors R and u, making wild guesses about them, and drawing conclusions. We do have a replication crisis on our hands. But the pre-study odds, whatever they are, are unchangeable. If this paper had been framed in terms of "the lower the pre-study odds, the higher the power will need to be to compensate to get an acceptable level of reproducibility", I would accept it wholeheartedly, although then it would have been a simple and obvious statement rather than a citation-grabber.
- AlexCoventry 9y ago
  Of course pre-study odds are low in fields querying complex systems on the basis of extremely weak theoretical frameworks.
  - 9y ago