It’s worth pointing out that processes like meta analyses ought to be able to catch #2. The problem is, GIGO. A lot of poorly designed studies (no real control

It’s worth pointing out that processes like meta analyses ought to be able to catch #2. The problem is, GIGO. A lot of poorly designed studies (no real control group, failure to control for other relevant variables, other methodological errors) make meta analysis outcomes unreliable. An area I follow closely is exercise science and I am amazed that researchers are able to get grants for some of the research they do. For instance, sometimes researchers aim to compare, say, the amount of hypertrophy on one training program versus another. They’ll use a study or maybe 15 individuals. The group on intervention A will experience 50% higher hypertrophy than intervention B, at p=0.08, and they’ll conclude that there’s no difference in hypertrophy between the two protocols rather than suggesting an increase in statistical power. Another great example is studies whose interventions fail to produce any muscle growth in new trainees. They’ll compare two programs, one with, say, higher training volume, and one with lower training volume. Both fail to produce results for some reason. They conclude that variable is not important, rather than perhaps concluding that their interventions are poorly designed since a population that is begging to gain muscle couldn’t gain anything from it.