4 ms·

The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed

The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling design). A hypothetical study of an infinitely large meta-population would produce the effect estimate of exactly 0.2, infinitely precisely. A series of very small studies will likely show quite divergent results, just on the basis of precision. A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect. If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable. There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)

lmeyerov 4y ago
Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.
- notafraudster 4y ago
  Both the authors of this study and meta-analysts generally have followup responses and other diagnostics and models that work under degraded assumptions, so I don't think it makes a ton of sense to respond to "this is the absolute most introductory example of why this kind of intuition works" with "it doesn't work in <XYZ> case". There are a variety of ways to think through what data looks like under publication bias or not irrespective of what the "correct" effect would be. I'd also add that this is a paper responding to an existing meta-analysis, so the claim that it's impossible to get a quantity of interest for a meta-analysis because the constituent studies are being inappropriately aggregated is itself an argument in favour of this paper's rejection of the original paper's finding. I personally just rejected an m-a at a social science journal on thursday because I thought it suffered from unaddressed garbage-in garbage-out along the lines you mention (non-experimental data, no attempt to pin down causal identification, inadequate qualitative discussion of risks of publication bias, unclear QOI), so I am sympathetic to the criticisms, but just know that there are next steps. :)
  - lmeyerov 4y ago
    it's one thing to reject a meta analysis, and i'm rarely suprised at that. i'm not a fan in general, as i've seen many break down when you dig in, even in seemingly 'nice' areas like whether medicine x works on disease y. it's another to reject the underlying phenomena because of metaanalysis.

lmeyerov 4y ago

Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.

notafraudster 4y ago
Both the authors of this study and meta-analysts generally have followup responses and other diagnostics and models that work under degraded assumptions, so I don't think it makes a ton of sense to respond to "this is the absolute most introductory example of why this kind of intuition works" with "it doesn't work in <XYZ> case". There are a variety of ways to think through what data looks like under publication bias or not irrespective of what the "correct" effect would be. I'd also add that this is a paper responding to an existing meta-analysis, so the claim that it's impossible to get a quantity of interest for a meta-analysis because the constituent studies are being inappropriately aggregated is itself an argument in favour of this paper's rejection of the original paper's finding. I personally just rejected an m-a at a social science journal on thursday because I thought it suffered from unaddressed garbage-in garbage-out along the lines you mention (non-experimental data, no attempt to pin down causal identification, inadequate qualitative discussion of risks of publication bias, unclear QOI), so I am sympathetic to the criticisms, but just know that there are next steps. :)
- lmeyerov 4y ago
  it's one thing to reject a meta analysis, and i'm rarely suprised at that. i'm not a fan in general, as i've seen many break down when you dig in, even in seemingly 'nice' areas like whether medicine x works on disease y. it's another to reject the underlying phenomena because of metaanalysis.