> there isn't a control group, and you haven't standardized it. Be very, very careful walking down this path. For example, case-control studies are inherently

> there isn't a control group, and you haven't standardized it. Be very, very careful walking down this path. For example, case-control studies are inherently non-representative of population rates and thus more appropriate for rare events (the tests reflect divergence from a large-sample approximation of the hypergeometric distribution to a chi-squared distribution). So if you want to test them against hypothetical nulls, it's better to permute existing data (provided it is in fact good data) or do your own replication study (focusing on the main effects) than to attempt to standardize a model that resists standardization. The question for many studies is not really 'how much' but 'whether', as in, whether a risk or a process or a mechanism explains a significant amount of the observed and unknown. One of the worst fallacies in science is the widespread belief that large-sample asymptotic results are applicable to smaller samples, often extreme-value distributions. You can't know what you don't know in some of these studies, in other words. You can attempt to build a model of what could have happened given the constraints under the null, and then permute the events in a simulation -- but for a meta-analysis with replication, how would you pull this off under a finite budget? And it would still be less expensive than your previous suggestion of attempting to directly replicate each of, say, 1000 studies. Even casting aside the intense reticence of researchers to perform purely replication-based studies, which is amazingly powerful to behold. So, while it is rather piecemeal and ad-hoc, my contention is that the current jury-rigged methodology is actually a less expensive way (albeit much slower) to converge at the best approximation to the truth in complex fields. You can disprove that a transcription factor will bind to a kinase after a functional mutation is introduced, because finding a control is easy -- you split a colony of cells, transfect one, don't transfect the other, and run everything in parallel. But when you are modeling genetic and environmental effects over a span of 25 years among 15,000 men and women, how do you go back in time and select an appropriate group of controls for meta-analysis? The truth is that you don't. Barring craven and institutionalized misinterpretation, such as is performed in some clinical trials analysis (when we all know better, and most biostatisticians can easily pick apart the errors if the information is made public), the current process is a slow, iterative, but useful approximation to the infinite-budget approach you propose. It might work for rinky-dink biochemistry or psychology experiments, or perhaps for microarray studies with expression signatures for things like tumor phenotypes. That's not too awful, and you could re-pilot the study for replication (happens a lot already). But the monster cohort studies that spawn sub-studies -- good luck with that! The big epidemiological studies are among the most mistreated of all, because carefully parametrized statements of summary results are then spun by talking heads to sound as dramatic as possible. Sometimes the principal investigators will get in on the game, but as often as not, calmly presented information ("we saw a 2.5x (95% ci: 1.4x-3.7x) increase in bladder cancer risk for GSTM1 null phenotypes exposed to N-nitrosamines") will be re-spun as "KILLER CANCER GENE FOUND BY PIONEERING RESEARCHERS AT THE UNIVERSITY OF SOUTH SCRANTON!!!1". Don't shoot the message, shoot the messenger, for those...