3 ms·

The claim in the post is that the replication crisis is real. Of course, it's among those you consider messageboard folk so you can dismiss it on those grounds,

The claim in the post is that the replication crisis is real. Of course, it's among those you consider messageboard folk so you can dismiss it on those grounds, but I think you are wrong to say it has nothing to do with what you wrote since it attempts to deal with both the numerator and the denominator by taking a sample across fields.

tptacek 2y ago
I don't know what point you're trying to make here. Is it that there are fraudulent research papers in journals? Everybody knows that's true. Is it that there are tens of thousands of them produced every year? That's made up. People are counting on readers not to know or take the time to learn how much research is happening, so that individual instances of the problem (in the United States) seem systemic.
- renewiltord 2y ago
  The post makes the point that a large amount of science produced in the US is not trustworthy. Fraudulence implies intentional deception, but it is more likely that it is simply a response to incentives. I don't see why you can't understand what I'm saying, but in the hope that it's easier I'll put it in bullet points and try to be direct * The post is a sample of papers across many fields * It describes a method to determine whether widespread p-hacking is occurring in a field * Widespread p-hacking in a field will lead to unreproducible results (because the results are an artifact of the data rather than reflecting any underlying cause) * You said: The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator. * The approach in the post uses a statistical method across many papers across many fields. It does not rely on the existence of some number of non-replicable results (in fact, one should expect some percentage of results to be non-replicable). * Therefore, it doesn't rely on "the numerator" so to speak, but a sample of all papers in a field to characterize the field's replicability. You're a smart guy. I follow your posts here and on Twitter. So I can only assume that either I've flipped some bozo bit or something in my communication style is abstruse. I don't understand why you're saying that p-hacking has nothing to do with the replication crisis (most would expect the two to be quite related), or that saying "the replication crisis is real" means I am claiming that individual papers exist that are non-replicable (which is fine, but not a replication crisis). In either case, we're clearly getting nothing out of talking to each other so I might as well leave off here.
  - tptacek 2y ago
    I think if the claims that article makes are both (i) true and (ii) meaningful (as opposed to an annoying artifact of the norms of how these fields operate) we'd be seeing downstream effects of it that we do not see†. I think this is a particularly weird time in the history of biomedical science to be trying to further the argument that everybody is adding cards to a giant house of cards, but you do you. Either way: people are responding to specific stories about fraudulent research, and my point is that the numerator of those stories is infinitesimal compared to the denominator of how many papers are published in these fields annually, by how many separately-funded R1 research teams. I'm really not interested in being boxed into a debate about whether p-hacking matters, or scientific fraud exists. Fraud is bad! I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. † That article is about a bunch of different fields, and I'm commenting only on a very specific subset of them; fuck do I know how well social science is operating right now
    - ziddoap 2y ago
      >I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. I've been trying to think about what you meant since you wrote this earlier in another comment, and I'm having some trouble wrapping my head around it so I think I'll just ask (if you'd be so kind to explain). What makes some level of fraud more optimal than zero fraud?
      - tptacek 2y ago
        Patrick McKenzie popularized that wording, but I think he may have taken inspiration from Dan Davies, and the idea is older still. It's simply that fraud is endemic to large systems, and past a certain threshold eliminating it makes people worse-off on balance as the countermeasures retard the efficient operation of the system. You can get to zero fraud in lending easily: just don't make loans. Grinding all scientific research to a halt to "true up" the statistical reporting used for otherwise valid and important research results would be a very good example of an intervention that would (a) "reduce fraud" and (b) leave us all significantly worse off.

tptacek 2y ago

I don't know what point you're trying to make here. Is it that there are fraudulent research papers in journals? Everybody knows that's true. Is it that there are tens of thousands of them produced every year? That's made up. People are counting on readers not to know or take the time to learn how much research is happening, so that individual instances of the problem (in the United States) seem systemic.

renewiltord 2y ago
The post makes the point that a large amount of science produced in the US is not trustworthy. Fraudulence implies intentional deception, but it is more likely that it is simply a response to incentives. I don't see why you can't understand what I'm saying, but in the hope that it's easier I'll put it in bullet points and try to be direct * The post is a sample of papers across many fields * It describes a method to determine whether widespread p-hacking is occurring in a field * Widespread p-hacking in a field will lead to unreproducible results (because the results are an artifact of the data rather than reflecting any underlying cause) * You said: The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator. * The approach in the post uses a statistical method across many papers across many fields. It does not rely on the existence of some number of non-replicable results (in fact, one should expect some percentage of results to be non-replicable). * Therefore, it doesn't rely on "the numerator" so to speak, but a sample of all papers in a field to characterize the field's replicability. You're a smart guy. I follow your posts here and on Twitter. So I can only assume that either I've flipped some bozo bit or something in my communication style is abstruse. I don't understand why you're saying that p-hacking has nothing to do with the replication crisis (most would expect the two to be quite related), or that saying "the replication crisis is real" means I am claiming that individual papers exist that are non-replicable (which is fine, but not a replication crisis). In either case, we're clearly getting nothing out of talking to each other so I might as well leave off here.
- tptacek 2y ago
  I think if the claims that article makes are both (i) true and (ii) meaningful (as opposed to an annoying artifact of the norms of how these fields operate) we'd be seeing downstream effects of it that we do not see†. I think this is a particularly weird time in the history of biomedical science to be trying to further the argument that everybody is adding cards to a giant house of cards, but you do you. Either way: people are responding to specific stories about fraudulent research, and my point is that the numerator of those stories is infinitesimal compared to the denominator of how many papers are published in these fields annually, by how many separately-funded R1 research teams. I'm really not interested in being boxed into a debate about whether p-hacking matters, or scientific fraud exists. Fraud is bad! I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. † That article is about a bunch of different fields, and I'm commenting only on a very specific subset of them; fuck do I know how well social science is operating right now
  - ziddoap 2y ago
    >I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. I've been trying to think about what you meant since you wrote this earlier in another comment, and I'm having some trouble wrapping my head around it so I think I'll just ask (if you'd be so kind to explain). What makes some level of fraud more optimal than zero fraud?
    - tptacek 2y ago
      Patrick McKenzie popularized that wording, but I think he may have taken inspiration from Dan Davies, and the idea is older still. It's simply that fraud is endemic to large systems, and past a certain threshold eliminating it makes people worse-off on balance as the countermeasures retard the efficient operation of the system. You can get to zero fraud in lending easily: just don't make loans. Grinding all scientific research to a halt to "true up" the statistical reporting used for otherwise valid and important research results would be a very good example of an intervention that would (a) "reduce fraud" and (b) leave us all significantly worse off.