5 ms·

Actually, you're wrong about that. According to this Cochrane systematic review, RCT's and observational studies are usually in close agreement: https://www.coc

Actually, you're wrong about that. According to this Cochrane systematic review, RCT's and observational studies are usually in close agreement: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000034.pub2/full https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR...

rich_sasha 4y ago
Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs. Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).
Gatsky 4y ago
Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.
- vcdimension 4y ago
  @Gatsky: ah I see, yes that sounds highly plausible, but can you supply any evidence of this, or is it just hearsay?
gojomo 4y ago
Such a review is necessarily limited to topics which have both an RCT and an observational study. As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement. (It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.) The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.
- vcdimension 4y ago
  @gojomo: Isn't it normally the case that an RCT follows an observational study? There's not much point in doing an observational study if someone's already done an RCT, but there is the other way round, since RCT's are more trusted. Wrt publication bias; an underestimation of the false positive rate of observational studies wrt RCTs could occur if non-significant RCTs are being held back at a higher rate than observational ones, but it seems more likely to me that it's the other way around since RCTs are generally more expensive and time consuming than observational studies, and more trusted, so it's a bigger loss if the result isn't published.
  - gojomo 4y ago
    I don’t have a sense of the ‘normal’; it might vary by field. For example, for COVID vaccines, we’re being inundated with observational data - often informal, sometimes in reviewed studies - long after initial approval RCTs.

rich_sasha 4y ago

Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs. Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).

Gatsky 4y ago

Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.

vcdimension 4y ago
@Gatsky: ah I see, yes that sounds highly plausible, but can you supply any evidence of this, or is it just hearsay?

gojomo 4y ago

Such a review is necessarily limited to topics which have both an RCT and an observational study. As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement. (It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.) The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.

vcdimension 4y ago
@gojomo: Isn't it normally the case that an RCT follows an observational study? There's not much point in doing an observational study if someone's already done an RCT, but there is the other way round, since RCT's are more trusted. Wrt publication bias; an underestimation of the false positive rate of observational studies wrt RCTs could occur if non-significant RCTs are being held back at a higher rate than observational ones, but it seems more likely to me that it's the other way around since RCTs are generally more expensive and time consuming than observational studies, and more trusted, so it's a bigger loss if the result isn't published.
- gojomo 4y ago
  I don’t have a sense of the ‘normal’; it might vary by field. For example, for COVID vaccines, we’re being inundated with observational data - often informal, sometimes in reviewed studies - long after initial approval RCTs.