Indeed! I have a particularly relevant horror story. For one of my graduate classes, I built a game where two people (a liar and a truth teller) would try to c

Indeed! I have a particularly relevant horror story. For one of my graduate classes, I built a game where two people (a liar and a truth teller) would try to convince a third person (the judge) they had experienced something. The goal was to see if people used different language when lying than they did when telling the truth. It was a fun project and we were focused more on the NLP/ML side of things. Turns out to be reasonably possible to separate our ~500 examples via a linear SVM, with some interesting separating words. That's where our claims stopped. Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed "Oh look! Pronouns are statistically significant! Oh and possessive nouns too!" -- I cringed. The flip side of this is that as statisticians, we do know how to handle different kinds of testing. If you're going to be looking at all these different outcomes, that's fine, but you just need to correct for it. We've had Bonferroni correction since the 1960s; Benjamini-Hochberg and related false discovery rate methods have been around for almost 20 years now. In fact, there are even situations where the data can help define a prior for your hypothesis testing [1]. Lastly, there is this quote from the article: > There is no statistical quality board that could enforce such larger analyses—nor would we believe such coercion to be appropriate I'm not sure if that's such a bad idea. In theory, that should be the job of the journal reviewers and editor. In practice, it's often the blind leading the blind, with confirmation bias thrown in to boot. Maybe we need an SRB as a companion to IRB. [1] http://arxiv.org/abs/1411.6144 http://arxiv.org/abs/1411.6144