4 ms·

This is rather expensive, and for human studies, would require both extremely fine control of bias (very, very difficult in meta-analyses due to publication bia

by apathy 19y ago

This is rather expensive, and for human studies, would require both extremely fine control of bias (very, very difficult in meta-analyses due to publication bias) as well as extremely well-established investigators who could get a hugely negative study published in a highly visible journal. Most academics know better than to challenge the norms. Corporate control of scholarly journals (in terms of the actual distribution and management of the content rights) has not served the actual practice of science very well. In any event, this massive meta-replication study you speak of is done, very slowly and piecemeal, every day in every reasonable field. As new studies fail to replicate old studies, the conclusions of the irreproducible are deprecated, while those that consistently bear fruit are entrenched. Given complex models and expensive studies (eg. massive multi-year cohort studies of genetic, environmental, and gene*environment interaction effects on cancer risk), that's about all you can hope for. Science is simultaneously practiced in a manner worse than anyone's fears (on the level of individual studies) and better than anyone's hopes (on the level of accretion of knowledge over time vs. budgetary constraints). The fraudsters burn out quickly in most rigorous fields.

byrneseyeview 19y ago
>In any event, this massive meta-replication study you speak of is done, very slowly and piecemeal, every day in every reasonable field. As new studies fail to replicate old studies, the conclusions of the irreproducible are deprecated, while those that consistently bear fruit are entrenched. Given complex models and expensive studies (eg. massive multi-year cohort studies of genetic, environmental, and gene*environment interaction effects on cancer risk), that's about all you can hope for. Sure, but in a very ad hoc way. It's not a study if you're just sucking up data when you find it -- there isn't a control group, and you haven't standardized it.
- apathy 19y ago
  > there isn't a control group, and you haven't standardized it. Be very, very careful walking down this path. For example, case-control studies are inherently non-representative of population rates and thus more appropriate for rare events (the tests reflect divergence from a large-sample approximation of the hypergeometric distribution to a chi-squared distribution). So if you want to test them against hypothetical nulls, it's better to permute existing data (provided it is in fact good data) or do your own replication study (focusing on the main effects) than to attempt to standardize a model that resists standardization. The question for many studies is not really 'how much' but 'whether', as in, whether a risk or a process or a mechanism explains a significant amount of the observed and unknown. One of the worst fallacies in science is the widespread belief that large-sample asymptotic results are applicable to smaller samples, often extreme-value distributions. You can't know what you don't know in some of these studies, in other words. You can attempt to build a model of what could have happened given the constraints under the null, and then permute the events in a simulation -- but for a meta-analysis with replication, how would you pull this off under a finite budget? And it would still be less expensive than your previous suggestion of attempting to directly replicate each of, say, 1000 studies. Even casting aside the intense reticence of researchers to perform purely replication-based studies, which is amazingly powerful to behold. So, while it is rather piecemeal and ad-hoc, my contention is that the current jury-rigged methodology is actually a less expensive way (albeit much slower) to converge at the best approximation to the truth in complex fields. You can disprove that a transcription factor will bind to a kinase after a functional mutation is introduced, because finding a control is easy -- you split a colony of cells, transfect one, don't transfect the other, and run everything in parallel. But when you are modeling genetic and environmental effects over a span of 25 years among 15,000 men and women, how do you go back in time and select an appropriate group of controls for meta-analysis? The truth is that you don't. Barring craven and institutionalized misinterpretation, such as is performed in some clinical trials analysis (when we all know better, and most biostatisticians can easily pick apart the errors if the information is made public), the current process is a slow, iterative, but useful approximation to the infinite-budget approach you propose. It might work for rinky-dink biochemistry or psychology experiments, or perhaps for microarray studies with expression signatures for things like tumor phenotypes. That's not too awful, and you could re-pilot the study for replication (happens a lot already). But the monster cohort studies that spawn sub-studies -- good luck with that! The big epidemiological studies are among the most mistreated of all, because carefully parametrized statements of summary results are then spun by talking heads to sound as dramatic as possible. Sometimes the principal investigators will get in on the game, but as often as not, calmly presented information ("we saw a 2.5x (95% ci: 1.4x-3.7x) increase in bladder cancer risk for GSTM1 null phenotypes exposed to N-nitrosamines") will be re-spun as "KILLER CANCER GENE FOUND BY PIONEERING RESEARCHERS AT THE UNIVERSITY OF SOUTH SCRANTON!!!1". Don't shoot the message, shoot the messenger, for those...

by apathy 19y ago

byrneseyeview 19y ago

>In any event, this massive meta-replication study you speak of is done, very slowly and piecemeal, every day in every reasonable field. As new studies fail to replicate old studies, the conclusions of the irreproducible are deprecated, while those that consistently bear fruit are entrenched. Given complex models and expensive studies (eg. massive multi-year cohort studies of genetic, environmental, and gene*environment interaction effects on cancer risk), that's about all you can hope for. Sure, but in a very ad hoc way. It's not a study if you're just sucking up data when you find it -- there isn't a control group, and you haven't standardized it.

apathy 19y ago
> there isn't a control group, and you haven't standardized it. Be very, very careful walking down this path. For example, case-control studies are inherently non-representative of population rates and thus more appropriate for rare events (the tests reflect divergence from a large-sample approximation of the hypergeometric distribution to a chi-squared distribution). So if you want to test them against hypothetical nulls, it's better to permute existing data (provided it is in fact good data) or do your own replication study (focusing on the main effects) than to attempt to standardize a model that resists standardization. The question for many studies is not really 'how much' but 'whether', as in, whether a risk or a process or a mechanism explains a significant amount of the observed and unknown. One of the worst fallacies in science is the widespread belief that large-sample asymptotic results are applicable to smaller samples, often extreme-value distributions. You can't know what you don't know in some of these studies, in other words. You can attempt to build a model of what could have happened given the constraints under the null, and then permute the events in a simulation -- but for a meta-analysis with replication, how would you pull this off under a finite budget? And it would still be less expensive than your previous suggestion of attempting to directly replicate each of, say, 1000 studies. Even casting aside the intense reticence of researchers to perform purely replication-based studies, which is amazingly powerful to behold. So, while it is rather piecemeal and ad-hoc, my contention is that the current jury-rigged methodology is actually a less expensive way (albeit much slower) to converge at the best approximation to the truth in complex fields. You can disprove that a transcription factor will bind to a kinase after a functional mutation is introduced, because finding a control is easy -- you split a colony of cells, transfect one, don't transfect the other, and run everything in parallel. But when you are modeling genetic and environmental effects over a span of 25 years among 15,000 men and women, how do you go back in time and select an appropriate group of controls for meta-analysis? The truth is that you don't. Barring craven and institutionalized misinterpretation, such as is performed in some clinical trials analysis (when we all know better, and most biostatisticians can easily pick apart the errors if the information is made public), the current process is a slow, iterative, but useful approximation to the infinite-budget approach you propose. It might work for rinky-dink biochemistry or psychology experiments, or perhaps for microarray studies with expression signatures for things like tumor phenotypes. That's not too awful, and you could re-pilot the study for replication (happens a lot already). But the monster cohort studies that spawn sub-studies -- good luck with that! The big epidemiological studies are among the most mistreated of all, because carefully parametrized statements of summary results are then spun by talking heads to sound as dramatic as possible. Sometimes the principal investigators will get in on the game, but as often as not, calmly presented information ("we saw a 2.5x (95% ci: 1.4x-3.7x) increase in bladder cancer risk for GSTM1 null phenotypes exposed to N-nitrosamines") will be re-spun as "KILLER CANCER GENE FOUND BY PIONEERING RESEARCHERS AT THE UNIVERSITY OF SOUTH SCRANTON!!!1". Don't shoot the message, shoot the messenger, for those...