3 ms·
Aside from the matter of whether the article's criticisms of the paper in question are valid[1], I think there's some equivocation going on here between two kin
by gjm11 3y ago
Aside from the matter of whether the article's criticisms of the paper in question are valid[1], I think there's some equivocation going on here between two kinds of thing that can both be called failures of "replicability" or "reproducibility" but that are very different. (1) Sometimes a paper doesn't make it perfectly clear exactly what the authors did and why. (2) Sometimes it's clear enough that you can try to do the same thing, and when you do you get different results.
[1] As others have pointed out, there are rebuttals in the comments from some of the authors of the paper; make of them what you will.
The "reproducibility crisis" / "replication crisis" is about #2: lots of published work, it seems, reports results that other people trying to do the same experiments can't reproduce. This probably means that their results, or at least a lot of their results, are wrong, which is potentially a big deal.
The article's complaint about the paper by van Zwet, Gelman et al isn't that. It's that some details of the statistical analysis aren't reproducible in sense 1: the authors did some pruning of the data but apparently didn't explain exactly what criteria they used for the pruning.
You could argue that actually #1 is worse than #2, because maybe what the authors did is bad but you can't even tell. Or you could argue that it's not as bad as #2, because most of the time (or: in this specific case) what they do is OK. But I don't think it makes any sense to suggest that they're the same thing, that there's some great irony if a paper about #2 has a #1 problem. That's just making a pun on two completely different sorts of reproducibility.
(Note 1: The article makes another complaint too, which if valid might be a big deal, but it doesn't have anything much to do with reproducibility in either sense.)
(Note 2: One of the authors of the paper does explain, in comments on the OP, exactly how they did the pruning. I have not checked what the original paper says about that; it doesn't seem to be available freely online.)
(Note 3: Even if the Zwet/Gelman/... paper were absolute trash, I don't really see how that would make the reproducibility crisis irreproducible. This is one paper published in 2023; the reproducibility crisis has been going on for years and involves many many unsuccessful attempts at replication.)