3 ms·
Such a review is necessarily limited to topics which have both an RCT and an observational study. As such, the general bias against publishing non-significant
by gojomo 4y ago
Such a review is necessarily limited to topics which have both an RCT and an observational study.
As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement.
(It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.)
The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.
- vcdimension 4y ago@gojomo: Isn't it normally the case that an RCT follows an observational study? There's not much point in doing an observational study if someone's already done an RCT, but there is the other way round, since RCT's are more trusted. Wrt publication bias; an underestimation of the false positive rate of observational studies wrt RCTs could occur if non-significant RCTs are being held back at a higher rate than observational ones, but it seems more likely to me that it's the other way around since RCTs are generally more expensive and time consuming than observational studies, and more trusted, so it's a bigger loss if the result isn't published.
- gojomo 4y agoI don’t have a sense of the ‘normal’; it might vary by field. For example, for COVID vaccines, we’re being inundated with observational data - often informal, sometimes in reviewed studies - long after initial approval RCTs.