I recently read the book "Common Errors in Statistics." It was really illuminating. They spent a particularly long time discussing errors in various medical stu

I recently read the book "Common Errors in Statistics." It was really illuminating. They spent a particularly long time discussing errors in various medical studies, although I think that's because the authors' backgrounds are in that area rather than medicine having particularly bad statistics. To attempt to summarize the book, it seems that statistical errors tend to boil down to one of three problems: * Improperly building the experiment, failing to ensure independence or account for confounding variables. * Misunderstanding significance. Reading too much meaning into p-values, and performing post-hoc analysis. Remember that a 95 percent significance level means that in 5 percent of the trials, you'll get a positive result by random chance. * Improperly using tests, by failing to account for their limitations. For instance, using a test that assumes normality when you don't have a strong case that the data is normal. Or using a test that requires a certain sample size on a dataset that is too small. When you think about it, most scientists receive a few semesters of statistics strung out through their education. They learn a little about probability, then how to do a few significance tests and something here and there about regression. They then start doing work where in pretty much every field nowadays, publishing a paper involves statistical analysis at some point. And most of them grope their way through it based on hazy memories of introductory stats classes taken years before.