3 ms·

The post makes the point that a large amount of science produced in the US is not trustworthy. Fraudulence implies intentional deception, but it is more likely

The post makes the point that a large amount of science produced in the US is not trustworthy. Fraudulence implies intentional deception, but it is more likely that it is simply a response to incentives. I don't see why you can't understand what I'm saying, but in the hope that it's easier I'll put it in bullet points and try to be direct * The post is a sample of papers across many fields * It describes a method to determine whether widespread p-hacking is occurring in a field * Widespread p-hacking in a field will lead to unreproducible results (because the results are an artifact of the data rather than reflecting any underlying cause) * You said: The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator. * The approach in the post uses a statistical method across many papers across many fields. It does not rely on the existence of some number of non-replicable results (in fact, one should expect some percentage of results to be non-replicable). * Therefore, it doesn't rely on "the numerator" so to speak, but a sample of all papers in a field to characterize the field's replicability. You're a smart guy. I follow your posts here and on Twitter. So I can only assume that either I've flipped some bozo bit or something in my communication style is abstruse. I don't understand why you're saying that p-hacking has nothing to do with the replication crisis (most would expect the two to be quite related), or that saying "the replication crisis is real" means I am claiming that individual papers exist that are non-replicable (which is fine, but not a replication crisis). In either case, we're clearly getting nothing out of talking to each other so I might as well leave off here.

tptacek 2y ago
I think if the claims that article makes are both (i) true and (ii) meaningful (as opposed to an annoying artifact of the norms of how these fields operate) we'd be seeing downstream effects of it that we do not see†. I think this is a particularly weird time in the history of biomedical science to be trying to further the argument that everybody is adding cards to a giant house of cards, but you do you. Either way: people are responding to specific stories about fraudulent research, and my point is that the numerator of those stories is infinitesimal compared to the denominator of how many papers are published in these fields annually, by how many separately-funded R1 research teams. I'm really not interested in being boxed into a debate about whether p-hacking matters, or scientific fraud exists. Fraud is bad! I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. † That article is about a bunch of different fields, and I'm commenting only on a very specific subset of them; fuck do I know how well social science is operating right now
- ziddoap 2y ago
  >I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. I've been trying to think about what you meant since you wrote this earlier in another comment, and I'm having some trouble wrapping my head around it so I think I'll just ask (if you'd be so kind to explain). What makes some level of fraud more optimal than zero fraud?
  - tptacek 2y ago
    Patrick McKenzie popularized that wording, but I think he may have taken inspiration from Dan Davies, and the idea is older still. It's simply that fraud is endemic to large systems, and past a certain threshold eliminating it makes people worse-off on balance as the countermeasures retard the efficient operation of the system. You can get to zero fraud in lending easily: just don't make loans. Grinding all scientific research to a halt to "true up" the statistical reporting used for otherwise valid and important research results would be a very good example of an intervention that would (a) "reduce fraud" and (b) leave us all significantly worse off.
    - defrost 2y ago
      I'm in agreement with tptacek and P.McKenzie and feel it's something more generally true; perfection is the enemy of throughput. Whether it's mine processing, agriculture, bank lending, { insert domain } there are always a number of drags on profit ( fraud, equipment wear and tear, petty theft, holes in fences ) and generally always a threshold past which the cost of pursuit of perfection exceeds the decreasing marginal benefit.
    - ziddoap 2y ago
      That makes a lot more sense than the literal interpretation. Thanks.
      - s1artibartfast 2y ago
        I think that is the literal interpretation. That is to say, Zero is the optimal in an imaginary system, where cows are spherical and you can make all the assumptions you want. Once you start working with real world systems with more and complex constraints, the simple story goes away, and perfection is not possible. The phrasing challenges the reader to question the model with which they are using to define optimal.

tptacek 2y ago

I think if the claims that article makes are both (i) true and (ii) meaningful (as opposed to an annoying artifact of the norms of how these fields operate) we'd be seeing downstream effects of it that we do not see†. I think this is a particularly weird time in the history of biomedical science to be trying to further the argument that everybody is adding cards to a giant house of cards, but you do you. Either way: people are responding to specific stories about fraudulent research, and my point is that the numerator of those stories is infinitesimal compared to the denominator of how many papers are published in these fields annually, by how many separately-funded R1 research teams. I'm really not interested in being boxed into a debate about whether p-hacking matters, or scientific fraud exists. Fraud is bad! I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. † That article is about a bunch of different fields, and I'm commenting only on a very specific subset of them; fuck do I know how well social science is operating right now

ziddoap 2y ago
>I'll just keep reiterating that the optimal level of fraud in a large system is almost never zero. I've been trying to think about what you meant since you wrote this earlier in another comment, and I'm having some trouble wrapping my head around it so I think I'll just ask (if you'd be so kind to explain). What makes some level of fraud more optimal than zero fraud?
- tptacek 2y ago
  Patrick McKenzie popularized that wording, but I think he may have taken inspiration from Dan Davies, and the idea is older still. It's simply that fraud is endemic to large systems, and past a certain threshold eliminating it makes people worse-off on balance as the countermeasures retard the efficient operation of the system. You can get to zero fraud in lending easily: just don't make loans. Grinding all scientific research to a halt to "true up" the statistical reporting used for otherwise valid and important research results would be a very good example of an intervention that would (a) "reduce fraud" and (b) leave us all significantly worse off.
  - defrost 2y ago
    I'm in agreement with tptacek and P.McKenzie and feel it's something more generally true; perfection is the enemy of throughput. Whether it's mine processing, agriculture, bank lending, { insert domain } there are always a number of drags on profit ( fraud, equipment wear and tear, petty theft, holes in fences ) and generally always a threshold past which the cost of pursuit of perfection exceeds the decreasing marginal benefit.
  - ziddoap 2y ago
    That makes a lot more sense than the literal interpretation. Thanks.
    - s1artibartfast 2y ago
      I think that is the literal interpretation. That is to say, Zero is the optimal in an imaginary system, where cows are spherical and you can make all the assumptions you want. Once you start working with real world systems with more and complex constraints, the simple story goes away, and perfection is not possible. The phrasing challenges the reader to question the model with which they are using to define optimal.