3 ms·
Did anyone else read the statistical analysis section? There were some "interesting"/unclear choices made, but I'm going to pick on one in particular that could
by ramblenode 10y ago
Did anyone else read the statistical analysis section? There were some "interesting"/unclear choices made, but I'm going to pick on one in particular that could affect the outcome: the decision on how to weight the individual cohort studies on which this study is based (pg 6).
> A random-effects model was used to obtain a single pooled estimate of the HRs from the individual studies for each category. The study-specific HRs were weighted by the inverse of the sum of their variance and the estimated between-studies variance component.
First note that there is no citation which motivates this weighting scheme. There may be a valid reason for doing it this way, but that's not clear from the paper.
Second, "the sum of their variance" I take to mean the sum of the variance components of the regression model. That's interesting because the components mean very different things. In the simple case of least squares, there's the "explained variance": the "strength" of the relationship between Y and X, and there's also the residual variance: the variance in X that isn't explained by Y. Aside from the fact that they mean different things, their sum is not a constant and so are not directly comparable across models/studies.
Third, weighted by the "between-studies variance component" has the effect of down-weighting studies that show a large difference from the mean study result and up-weighting studies which are similar. Now, what would you say if I conducted an experiment and transformed the raw data points according to this same procedure? "That's not fair because that's not the data we collected!"[0] And herein lies the problem: the purpose of this meta-analysis is to assess whether we can draw any consistent conclusions from the results of the individual cohort studies, but by weighting studies toward the mean the authors are taking it as a forgone conclusion that the mean study is ground truth.
I'm by no means an expert on meta-analysis so someone please chime in if you think I'm off base, but this all strikes me as a very peculiar and unjustified set of steps being taken before the model is even fit.
[0] Yes, there are cases for data transformation, motivated either by theory or meeting regression assumptions, but this is neither.