4 ms·

Let me just say this is why "show me the study" is never a productive line of debate. If someone agrees with the point the study seems to support, they say, "H

Let me just say this is why "show me the study" is never a productive line of debate. If someone agrees with the point the study seems to support, they say, "Hah! Studies!" and call anyone who disagrees "anti-science". But if someone disagrees with the point, they pull up the study, pick out some section of it where any type of subjective judgment call was made (usually the details of the sample, because that's the most clearly subjective thing, making it the easiest thing to criticize), and say "There are real problems with this study, how about a real study? [that is, a study that agrees with my point]". Also popular is "Yes, but this author is affiliated with former employer x, y, or z". This is true of all sides, all the time. Very rarely do you see a study actually impact anyone's opinion about an important topic. It is usually only the impression that more studies support a specific position (that is, "the consensus", aka social pressure to appear studious and informed, which affects academics as much or more than it affects non-academics), that does it.

bayonetz 9y ago
Given the power we see in 100+ study meta-analysis in medicine, I think we have to move towards a model where research grants in other fields (psych, sociology, etc.) are jointly awarded to multiple independent groups to jumpstart the meta-analysis for a given new topic from the outset. How to keep the "independent" groups from colluding will be a challenge unfortunately. As will funding for the extra replications. For the reasons you cite, single, stand-alone studies are almost useless these days.
tshadley 9y ago
> But if someone disagrees with the point, they pull up the study, pick out some section of it where any type of subjective judgment call was made (usually the details of the sample, because that's the most clearly subjective thing, making it the easiest thing to criticize), and say "There are real problems with this study, how about a real study? [that is, a study that agrees with my point]". Also popular is "Yes, but this author is affiliated with former employer x, y, or z". This is virtually impossible to do effectively unless one is an expert in the field. Studies that have been published by reputable journals have gone through a rigorous peer review process. A layperson criticizing such a study is highly unlikely to discover any valid points that have escaped the experts.

bayonetz 9y ago

Given the power we see in 100+ study meta-analysis in medicine, I think we have to move towards a model where research grants in other fields (psych, sociology, etc.) are jointly awarded to multiple independent groups to jumpstart the meta-analysis for a given new topic from the outset. How to keep the "independent" groups from colluding will be a challenge unfortunately. As will funding for the extra replications. For the reasons you cite, single, stand-alone studies are almost useless these days.

tshadley 9y ago

> But if someone disagrees with the point, they pull up the study, pick out some section of it where any type of subjective judgment call was made (usually the details of the sample, because that's the most clearly subjective thing, making it the easiest thing to criticize), and say "There are real problems with this study, how about a real study? [that is, a study that agrees with my point]". Also popular is "Yes, but this author is affiliated with former employer x, y, or z". This is virtually impossible to do effectively unless one is an expert in the field. Studies that have been published by reputable journals have gone through a rigorous peer review process. A layperson criticizing such a study is highly unlikely to discover any valid points that have escaped the experts.