3 ms·

“doesn’t replicate then it’s almost certainly nonsense” Disagree. A significant finding is expected to occur due to chance in direct proportion to the surface

“doesn’t replicate then it’s almost certainly nonsense” Disagree. A significant finding is expected to occur due to chance in direct proportion to the surface area for such possibilities. More studies, more forking paths within those studies, more models, all increase the frequency of spurious findings. So one can do everything perfectly and still get “garbage out”. It is certain.

pmichaud 5y ago
I’m not super sure I understand your point, but I think you’re saying that it’s possible to run a good replication attempt on a good study and still have it not replicate. I agree with that. I’m not super sure how to correctly estimate the chance of that happening, but one dumb way I can think of is just using p value, so if it was .05 then you have a 1/20 chance of failing to replicate a study even if everything was done correctly. However, when I said “doesn’t replicate” I didn’t have a single attempt with a 5% chance of failure in mind. I had a field’s aggregate attempt to confirm the result in mind, which would include multiple attempts and checking for statistical bullshit and all that. Under those conditions the chances are vanishingly small of a whole field getting massively unlucky when trying to replicate a well-done study that theoretically should replicate. That’s what I had in mind, and I still think it’s right. — Rereading what you wrote, a different interpretation of what you said is that the original investigators might have done everything perfectly, and nevertheless found a significant result that was spurious just because that stuff can happen by chance. If that’s what you meant, I don’t understand the disagreement, except maybe semantically. I would call a perfectly done study that shows a spurious result “nonsense,” and I would expect replication attempts to show the result is nonsense, even if the process that generated the nonsense was perfect. Maybe you’re just saying you wouldn’t call a perfectly done study “nonsense,” regardless of the outcome?
- PeterisP 5y ago
  There's a bunch of disciplines (live animal experimentation, microorganisms and organic chemistry come to mind, though they aren't "my" fields) where it's genuinely difficult to perform experiments, where it's reasonably common for experimenter to screw up the experiment in multiple ways, and failure to replicate may just as well indicate not a flaw with the original experimenter but a weakness of the skills of the team trying to replicate. One could argue that it's a failure of not sufficiently detailed descriptions of the experiment, but it is how it is in different disciplines. A relevant example for that is a bunch of earlier machine learning research (current methods subjectively seem more robust in this regard) some of which were very difficult to replicate from scratch because it was very finicky and relied on many tiny details that simply can't all fit in the couple pages of a standard paper, but it was definitely not nonsense, because it could be replicated if you knew all the best practices from previous experience. I mean, providing code for a ML paper doesn't change whether the research results there are nonsense or not, but many people might have a very hard time replicating that research without the code.
- splithalf 5y ago
  Yes, a semantic disagreement: I don’t think of spurious findings as nonsense. Nonsense has negative connotations that I feel contributes to the real problem which is the competitive nature of research and misconceptions about “error.” Too often I hear people elevate the researcher in a sort of “great man” theory of scientific progress. Too rarely do I hear praise for those doing boring replication studies and the like. I worry that younger people hear someone published “nonsense” and think of it as an indication of the quality of the researcher. All of this leads to more p hacking, more avoidance of replication, and more inefficiency in the scientific process. The economic incentives are such that universities compete for big name researchers, those who don’t publish nonsense. That is the problem IMO.

pmichaud 5y ago

I’m not super sure I understand your point, but I think you’re saying that it’s possible to run a good replication attempt on a good study and still have it not replicate. I agree with that. I’m not super sure how to correctly estimate the chance of that happening, but one dumb way I can think of is just using p value, so if it was .05 then you have a 1/20 chance of failing to replicate a study even if everything was done correctly. However, when I said “doesn’t replicate” I didn’t have a single attempt with a 5% chance of failure in mind. I had a field’s aggregate attempt to confirm the result in mind, which would include multiple attempts and checking for statistical bullshit and all that. Under those conditions the chances are vanishingly small of a whole field getting massively unlucky when trying to replicate a well-done study that theoretically should replicate. That’s what I had in mind, and I still think it’s right. — Rereading what you wrote, a different interpretation of what you said is that the original investigators might have done everything perfectly, and nevertheless found a significant result that was spurious just because that stuff can happen by chance. If that’s what you meant, I don’t understand the disagreement, except maybe semantically. I would call a perfectly done study that shows a spurious result “nonsense,” and I would expect replication attempts to show the result is nonsense, even if the process that generated the nonsense was perfect. Maybe you’re just saying you wouldn’t call a perfectly done study “nonsense,” regardless of the outcome?

PeterisP 5y ago
There's a bunch of disciplines (live animal experimentation, microorganisms and organic chemistry come to mind, though they aren't "my" fields) where it's genuinely difficult to perform experiments, where it's reasonably common for experimenter to screw up the experiment in multiple ways, and failure to replicate may just as well indicate not a flaw with the original experimenter but a weakness of the skills of the team trying to replicate. One could argue that it's a failure of not sufficiently detailed descriptions of the experiment, but it is how it is in different disciplines. A relevant example for that is a bunch of earlier machine learning research (current methods subjectively seem more robust in this regard) some of which were very difficult to replicate from scratch because it was very finicky and relied on many tiny details that simply can't all fit in the couple pages of a standard paper, but it was definitely not nonsense, because it could be replicated if you knew all the best practices from previous experience. I mean, providing code for a ML paper doesn't change whether the research results there are nonsense or not, but many people might have a very hard time replicating that research without the code.
splithalf 5y ago
Yes, a semantic disagreement: I don’t think of spurious findings as nonsense. Nonsense has negative connotations that I feel contributes to the real problem which is the competitive nature of research and misconceptions about “error.” Too often I hear people elevate the researcher in a sort of “great man” theory of scientific progress. Too rarely do I hear praise for those doing boring replication studies and the like. I worry that younger people hear someone published “nonsense” and think of it as an indication of the quality of the researcher. All of this leads to more p hacking, more avoidance of replication, and more inefficiency in the scientific process. The economic incentives are such that universities compete for big name researchers, those who don’t publish nonsense. That is the problem IMO.