4 ms·

I wish more scientists studied meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis; https://en.wikipedia.org/wiki/Meta-analysis; an area of study specifi

by dj-wonk 11y ago

I wish more scientists studied meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis; https://en.wikipedia.org/wiki/Meta-analysis; an area of study specifically tasked with the challenge of synthesizing various study results. Roughly speaking, each study is treated as a (weighted) observation and then combined using statistics. To put it another way, interpreting empirical results requires (or should require) statistics from bottom to top.

astazangasta 11y ago
To put it yet another way, overemphasis on statistics is a huge part of the problem. It is extremely difficult to get statistics right. Am I applying a relevant test? Even if it is relevant, is it too sensitive? Is my dataset unbiased or have I controlled for all of the significant biases? Multiple test corrections are also some of the most useless statistical tests you can perform. They're basically only useful for reducing confidence to reasonable levels. Hundreds of bad tests cannot yield a useful scientific inference through sheer volume. Meta-analysis is interesting, but it's ultimately a shoddy attempt to patch over the much larger problem of widespread methodological failure. We can't do bad science for decades and then expect meta-analysis to yield new insights; at best it can serve to expose the poor quality of that work.
- dj-wonk 11y ago
  > Meta-analysis is interesting, but it's ultimately a shoddy attempt to patch over the much larger problem of widespread methodological failure. I think I know what you are getting at. Let me try to unpack what I mean, and I would like to see if you would agree... I agree that "widespread methodological failure" is a problem with science in the real world. However, by your writing, it would seem that you attack meta-analysis as a technique. 1. Meta-analysis (MA) is a solid technique and not shoddy. I'm not saying it works equally well in all situations, but I am saying that it works better than not using it. :) 2. MA cannot "correct" one underlying study that is flawed. (This is obvious; doing statistics with a sample size of one is silly.) 3. Nor can MA correct for a systemic problem in underlying studies. 4. However, in the same way that statistics can be resilient over random errors in measurement, MA can handle studies of varying quality, provided that they do not have significant errors pointing in the same directions. (This is the often claimed, but less-often tested, assumption that errors are randomly distributed.)

by dj-wonk 11y ago

astazangasta 11y ago

To put it yet another way, overemphasis on statistics is a huge part of the problem. It is extremely difficult to get statistics right. Am I applying a relevant test? Even if it is relevant, is it too sensitive? Is my dataset unbiased or have I controlled for all of the significant biases? Multiple test corrections are also some of the most useless statistical tests you can perform. They're basically only useful for reducing confidence to reasonable levels. Hundreds of bad tests cannot yield a useful scientific inference through sheer volume. Meta-analysis is interesting, but it's ultimately a shoddy attempt to patch over the much larger problem of widespread methodological failure. We can't do bad science for decades and then expect meta-analysis to yield new insights; at best it can serve to expose the poor quality of that work.

dj-wonk 11y ago
> Meta-analysis is interesting, but it's ultimately a shoddy attempt to patch over the much larger problem of widespread methodological failure. I think I know what you are getting at. Let me try to unpack what I mean, and I would like to see if you would agree... I agree that "widespread methodological failure" is a problem with science in the real world. However, by your writing, it would seem that you attack meta-analysis as a technique. 1. Meta-analysis (MA) is a solid technique and not shoddy. I'm not saying it works equally well in all situations, but I am saying that it works better than not using it. :) 2. MA cannot "correct" one underlying study that is flawed. (This is obvious; doing statistics with a sample size of one is silly.) 3. Nor can MA correct for a systemic problem in underlying studies. 4. However, in the same way that statistics can be resilient over random errors in measurement, MA can handle studies of varying quality, provided that they do not have significant errors pointing in the same directions. (This is the often claimed, but less-often tested, assumption that errors are randomly distributed.)