It saddens me and depresses me greatly regarding the job prospects of working with statistics in modern tech companies when I read articles like this. I know o

It saddens me and depresses me greatly regarding the job prospects of working with statistics in modern tech companies when I read articles like this. I know others will disagree with me, but my feeling after working in a number of jobs in which large systems were built to automate large-scale frequentist hypothesis testing operations is that these methods just wholly fail, and that business politics leads people to read whatever conclusions they'd like to from the results, and give credit back to the statistical framework even if it is not deserved. I should be careful not to over-state this too much. I understand there is a mathematical framework for these tools and despite being trained in machine learning and Bayesian methods, almost all of my professional experience has been working with frequentist statistics. Sometimes you have enough data that it would probably swamp out whatever prior you might choose; often you don't have (or want to invest in getting) expert priors, and "non-informative priors" and that ilk sort of rope in some of the same problems as what frequentism has. You can get a limited amount of evidence from doing simple frequentist things, but you also open the door for all of the misunderstandings of p-values, ignoring effect sizes, doing model selection by directly comparing t-stats, etc., which just seems to be way, way less of a problem if you use other methods. Some of the main sources of enlightenment for me on this have been: [0] http://lesswrong.com/lw/g13/against_nhst/ http://lesswrong.com/lw/g13/against_nhst/ [1] http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf [2] http://www.indiana.edu/~kruschke/articles/Kruschke2012JEPG.pdf http://www.indiana.edu/~kruschke/articles/Kruschke2012JEPG.p... The last article, by Kruschke, spends a lot of time on exactly the type of issue from the Twitter article -- something like two-group mean testing. In general, I cannot understand why people continue to use frequentist methods on policy problems -- that is, problems where you are inherently looking for Pr(Hypothesis | Data), and where, due to all of the many, many pitfalls from the above sources and elsewhere, you can be sure that Pr(Data | Hypothesis) is not at all a suitable proxy for Pr(Hypothesis | Data). I would view any labor costs associated with figuring out why A/A/B testing is not a useful idea to be pure waste, not because the math is wrong or anything (the math is interesting and good!) but because, from first principles, this just seems like totally and completely the wrong way to approach this kind of applied problem. And yet, everyone is doing it, and baking it right into their huge production analytics systems -- most notoriously all of the modern fly-by-night marketing tech startups.