Power analysis and confidence intervals have their own problems, but they're already a vast improvement over p-values. No need to go all-out Bayesian straight a

Power analysis and confidence intervals have their own problems, but they're already a vast improvement over p-values. No need to go all-out Bayesian straight away – though certainly, if we can get the software to a point where that would be possible for non-statisticians, that'd be awesome. (Bayes factors help when you don't want to fix a prior.) For anyone who's interested to read more about this middle ground approach, "The Essential Guide to Effect Sizes" by Paul Ellis is an awesome introduction not just to power analysis but to better statistical practice and statistical reporting that sticks to the frequentist tradition. Also, I don't feel subgroup analysis is necessarily the problem, at least not when you apply any of the available corrections for familywise error rates (Bonferroni and Tukey's HSD being two of the more well known). This often comes up in medical research, e.g. http://www.badscience.net/2009/04/a-frankly-thin-contrivance-for-writing-on-the-fascinating-issue-of-subgroup-analysis/ http://www.badscience.net/2009/04/a-frankly-thin-contrivance... -- but in those cases subgroup analyses are performed with bad intentions, a whole other problem! A bigger issue, perhaps, is drawing inferences from hypotheses that came up when analyzing the data in the first place – it's very appealing but it's also wonderfully circular in its logic and why in machine learning circles it's become almost self-evident to split up the data into training and test sets.