Power analysis and CI's should be elementary, but I would assert that they are actually not commonplace. Most people have a very surface-level understanding of

Power analysis and CI's should be elementary, but I would assert that they are actually not commonplace. Most people have a very surface-level understanding of the latter, and little understanding of the former. In my opinion, A/B Testing has actually done a great service to power analysis. I have seen many experiments in the academic world (social sciences are somewhat notorious for this) forgoing the power analysis for various reasons (fear: they would not be able to get the sample size needed for 80% power, inability to control sample size: you take whatever you can get with a convenience sample). As a statistician, I breathe a sigh of relief with the amount of emphasis power analysis receives in the A/B world. It's a step in the right direction (if you're an acolyte to the dark world of Neyman-Pearson). As for bringing better stats to high tech, I've thought of this as a wonderful challenge. I'd especially like to see more focus on not violating modeling assumptions (more non and semi-parametrics), and using some more modern techniques from the ML and Bayes literature. Hypothesis testing is so last century :). Would love to discuss it further with some similarly-inclined HN folks. Sorry for all the parentheticals. You'd think I was a lisp programmer with the amount of parenthesis I used.