6 ms·

I'll start by saying that if you're going to roast a paper that an econ nobel winner and one of the most famous and respected working statisticians put their na

I'll start by saying that if you're going to roast a paper that an econ nobel winner and one of the most famous and respected working statisticians put their names on, you probably want to turn down the volume and double check your claims a little more before hitting "post." A z score is not at all morally equivalent to a p-value. It's just a standardized measure. Converting measures to z-scores aids in interpretation. They also can aid estimation in some cases: using non-standard parameterization in Bayesian analysis is often crucial to get MCMC to accurately sample from the posterior distribution. Sure, you can take a z score and look at the area under the curve and come up with a p value. But you don't have to. In the referenced paper, they use z scores to be able to standardize the measures in the papers they draw from, so they're comparable. The author's other critiques of the paper seem reasonable. It's a problem with all meta analyses: the amount of work it takes to correctly interpret publishes papers and then take those results and aggregate them is herculean. To do it to over 20,000 is inevitably going to lead to some mistakes. That said, those mistakes may not be fatal to the analysis. Moreover, saying "no one knows what happened to those 11,285 studies" without checking in with the authors is completely unfair. The first author responded with the code showing exactly how they achieved that figure. Nothing mysterious. Andrew Gelman responded in the comments to him, as did the first author. I find their responses convincing.

polygamous_bat 3y ago
> Moreover, saying "no one knows what happened to those 11,285 studies" without checking in with the authors is completely unfair. The first author responded with the code showing exactly how they achieved that figure. Nothing mysterious. But that is the whole point! The methodology of how they dropped the 11,285 studies was not even told in the original paper, and even in the comments the author doesn't explain "why", just "how". Hence, I think it's completely fair to call it "irreproducible". The point of doing "reproducible science" is not that I write a paper, you email me asking how did I come up with my number, and I email you back an explanation. No! The important details should be in the paper already. You may do some magic on your dataset, and that's fair _as long as you detail what magic you did and why_, and given you can defend that practice in front of your peers. Otherwise what is the point of preaching "reproducible science" at all?
- maxbond 3y ago
  > [T]he author doesn't explain "why", just "how". The below seems like a "why" to me. > The criteria for selecting the data are an atttempt to get the primary efficacy outcome, and to ensure that each trial occurs only once in our dataset. The selection for |z|<20 is because such large z-values are extremely unlikely for trials that aim to test if the effect is zero.
  - polygamous_bat 3y ago
    Ah I am sorry, seems like a comment was posted later (Jan 9th) providing a justification of their criteria. However, the reply comment it was posted on wasn't there when I first read the article and the comments on Jan 8th.
- pocketsand 3y ago
  I think your critiques are fair enough and I think the authors would likely agree they should have been more proactive with the data sharing. Maybe it sounds like I'm parsing too much, but I nevertheless still think saying "no one knows what happened" is unfair. They know, they shared, and justified what they did when called on it with literally almost no delay. I agree they shouldn't need to be called on it. Anyone who's advised students or asked even presenting researchers such questions know that often people will literally not know what happened to all their data.
  - lusus_naturae 3y ago
    > Anyone who's advised students or asked even presenting researchers such questions know that often people will literally not know what happened to all their data. I am sorry that’s been your experience, maybe it varies by field and quality of research? Most people I’ve questioned have provided reasonable answers to their findings. I don’t understand why anything needs to be assumed in bad faith or shoddily done. It’s all a bit Dunning-Kruger to me where everyone assumes that everyone else is doing shoddy or bad work.
    - 3y ago
sesm 3y ago
Regarding your appeal to authority: high ranks in today’s scientific system is an incentive to defend the system.
- stevenAthompson 3y ago
  I'm not sure it qualifies as argumentum ab auctoritate to say that a group comprised of Nobel prize winner's and renowned statistics experts are far more likely to be right about statistics than a solitary compsci professor. Even if it does qualify, It's widely accepted that argument from authority is perfectly valid and often necessary when performing inductive reasoning. I might even say that if you are not expert enough to judge the matter yourself (or willing to expend enough effort) it should be your default presumption that the more qualified speakers are correct, even if you have legitimate questions about their potential motivations.
  - pocketsand 3y ago
    My issue here is that Imbens and Gelman are respected because they are good statisticians but also clear thinkers who have proved themselves dedicated to doing good, careful work. If you have major issues with work their name is on, I would think their reputation would at least lead reasonable people to contact them first before writing something which leads to hurt feelings and numerous comments and corrections and more posts on each end. This whole affair is a case in point. Several authors ended up responding with very reasonable responses, which the author then acknowledged. Then he writes another post, which is more measured and insightful. The original post would have in fact been a far better one if the author had first sought those responses and just addressed them all together at once. His points about metascience would not then be dragged down by back and forth in the comments that clearly got personal.
Epa095 3y ago
"You Come At the King,You Best Not Miss" ;-)
hnu123 3y ago
> I'll start by saying that if you're going to roast a paper that an econ nobel winner "By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's." -- Paul Krugman ( Nobel Prize Winner in Economics ) If you have to pathetically appeal to authority right from the start, you probably have no argument worth considering.
dylukes 3y ago
Small quibble: converting a measure to a z-score requires an assumption of normality and that you're considering a population. For a sample the equivalent is the t-statistic, which indeed IS very often used for p-values (with the ever popular t-test) and has a decently strict list of assumptions (which, like those of [O|W|G]LS are very frequently ignored).
- pocketsand 3y ago
  z-scores just require knowing a standard deviation and a mean. you only need an assumption of normality if you want to do things like assign p-values. of course, that is mostly how they are used.

polygamous_bat 3y ago

> Moreover, saying "no one knows what happened to those 11,285 studies" without checking in with the authors is completely unfair. The first author responded with the code showing exactly how they achieved that figure. Nothing mysterious. But that is the whole point! The methodology of how they dropped the 11,285 studies was not even told in the original paper, and even in the comments the author doesn't explain "why", just "how". Hence, I think it's completely fair to call it "irreproducible". The point of doing "reproducible science" is not that I write a paper, you email me asking how did I come up with my number, and I email you back an explanation. No! The important details should be in the paper already. You may do some magic on your dataset, and that's fair _as long as you detail what magic you did and why_, and given you can defend that practice in front of your peers. Otherwise what is the point of preaching "reproducible science" at all?

maxbond 3y ago
> [T]he author doesn't explain "why", just "how". The below seems like a "why" to me. > The criteria for selecting the data are an atttempt to get the primary efficacy outcome, and to ensure that each trial occurs only once in our dataset. The selection for |z|<20 is because such large z-values are extremely unlikely for trials that aim to test if the effect is zero.
- polygamous_bat 3y ago
  Ah I am sorry, seems like a comment was posted later (Jan 9th) providing a justification of their criteria. However, the reply comment it was posted on wasn't there when I first read the article and the comments on Jan 8th.
pocketsand 3y ago
I think your critiques are fair enough and I think the authors would likely agree they should have been more proactive with the data sharing. Maybe it sounds like I'm parsing too much, but I nevertheless still think saying "no one knows what happened" is unfair. They know, they shared, and justified what they did when called on it with literally almost no delay. I agree they shouldn't need to be called on it. Anyone who's advised students or asked even presenting researchers such questions know that often people will literally not know what happened to all their data.
- lusus_naturae 3y ago
  > Anyone who's advised students or asked even presenting researchers such questions know that often people will literally not know what happened to all their data. I am sorry that’s been your experience, maybe it varies by field and quality of research? Most people I’ve questioned have provided reasonable answers to their findings. I don’t understand why anything needs to be assumed in bad faith or shoddily done. It’s all a bit Dunning-Kruger to me where everyone assumes that everyone else is doing shoddy or bad work.
  - 3y ago

sesm 3y ago

Regarding your appeal to authority: high ranks in today’s scientific system is an incentive to defend the system.

stevenAthompson 3y ago
I'm not sure it qualifies as argumentum ab auctoritate to say that a group comprised of Nobel prize winner's and renowned statistics experts are far more likely to be right about statistics than a solitary compsci professor. Even if it does qualify, It's widely accepted that argument from authority is perfectly valid and often necessary when performing inductive reasoning. I might even say that if you are not expert enough to judge the matter yourself (or willing to expend enough effort) it should be your default presumption that the more qualified speakers are correct, even if you have legitimate questions about their potential motivations.
- pocketsand 3y ago
  My issue here is that Imbens and Gelman are respected because they are good statisticians but also clear thinkers who have proved themselves dedicated to doing good, careful work. If you have major issues with work their name is on, I would think their reputation would at least lead reasonable people to contact them first before writing something which leads to hurt feelings and numerous comments and corrections and more posts on each end. This whole affair is a case in point. Several authors ended up responding with very reasonable responses, which the author then acknowledged. Then he writes another post, which is more measured and insightful. The original post would have in fact been a far better one if the author had first sought those responses and just addressed them all together at once. His points about metascience would not then be dragged down by back and forth in the comments that clearly got personal.

Epa095 3y ago

"You Come At the King,You Best Not Miss" ;-)

hnu123 3y ago

> I'll start by saying that if you're going to roast a paper that an econ nobel winner "By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's." -- Paul Krugman ( Nobel Prize Winner in Economics ) If you have to pathetically appeal to authority right from the start, you probably have no argument worth considering.

dylukes 3y ago

Small quibble: converting a measure to a z-score requires an assumption of normality and that you're considering a population. For a sample the equivalent is the t-statistic, which indeed IS very often used for p-values (with the ever popular t-test) and has a decently strict list of assumptions (which, like those of [O|W|G]LS are very frequently ignored).

pocketsand 3y ago
z-scores just require knowing a standard deviation and a mean. you only need an assumption of normality if you want to do things like assign p-values. of course, that is mostly how they are used.