3 ms·

If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting thi

If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting this correctly(and I by no means am sure that I am), I infer that they are saying in a fair publishing environment you’d expect to see more results that show less decisive results, therefor the current set of results is likely biased. Couldn’t this bias also happen in the other direction? It sounds like they’re saying the results are too good and don’t match other scientific patterns of publishing results.

mcswell 4y ago
"Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where publication of negative results is more likely than publication of positive results, although there are obviously heated debates over some results. Whether publication bias is the explanation in this example, I don't pretend to know.
- mikkergp 4y ago
  I guess the question would be are positive results missing for another reason, like they are harder to test for, therefor the data looks better than it should because in the aggregate they are better, but yeah this is probably unlikely.
- ghaff 4y ago
  There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons. But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a lot of people don't go with the default. We probably also shouldn't be surprised if a lot of people maybe wouldn't go with a maxed out default. Defaults wouldn't be nearly so powerful if they weren't typically chosen to be fairly reasonable for the average person in the target audience.
  - mikkergp 4y ago
    But I don’t think the goal is “most people can be nudged” so much as “a nudge is a cheap way to increase a behavior by 5-10% which is probably quite significant in policy circles. Low single digit percentage increase in people who pay their taxes would be huge.
notafraudster 4y ago
The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling design). A hypothetical study of an infinitely large meta-population would produce the effect estimate of exactly 0.2, infinitely precisely. A series of very small studies will likely show quite divergent results, just on the basis of precision. A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect. If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable. There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)
- lmeyerov 4y ago
  Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.

mcswell 4y ago

"Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where publication of negative results is more likely than publication of positive results, although there are obviously heated debates over some results. Whether publication bias is the explanation in this example, I don't pretend to know.

mikkergp 4y ago
I guess the question would be are positive results missing for another reason, like they are harder to test for, therefor the data looks better than it should because in the aggregate they are better, but yeah this is probably unlikely.
ghaff 4y ago
There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons. But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a lot of people don't go with the default. We probably also shouldn't be surprised if a lot of people maybe wouldn't go with a maxed out default. Defaults wouldn't be nearly so powerful if they weren't typically chosen to be fairly reasonable for the average person in the target audience.
- mikkergp 4y ago
  But I don’t think the goal is “most people can be nudged” so much as “a nudge is a cheap way to increase a behavior by 5-10% which is probably quite significant in policy circles. Low single digit percentage increase in people who pay their taxes would be huge.

notafraudster 4y ago

The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling design). A hypothetical study of an infinitely large meta-population would produce the effect estimate of exactly 0.2, infinitely precisely. A series of very small studies will likely show quite divergent results, just on the basis of precision. A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect. If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable. There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)

lmeyerov 4y ago
Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.