3 ms·
> In general, I cannot understand why people continue to use frequentist methods on policy problems -- that is, problems where you are inherently looking for Pr
by gwern 11y ago
> In general, I cannot understand why people continue to use frequentist methods on policy problems -- that is, problems where you are inherently looking for Pr(Hypothesis | Data), and where, due to all of the many, many pitfalls from the above sources and elsewhere, you can be sure that Pr(Data | Hypothesis) is not at all a suitable proxy for Pr(Hypothesis | Data).
To some extent, this is inherent to the application of the tools. In an A/B test, done by a business, it is incomplete to apply 'just' a frequentist hypothesis test (whether or not you fall into any of the traps) or just a Bayesian test with a relatively arbitrary threshold like 99% probability of improvement. As has been pointed out many times, your standard A/B test is focused solely on precisely whether there is an effect or not, while ignoring all the costs and benefits; which means you get precise expensive estimates of the wrong thing. You need to broaden the picture and explicitly acknowledge that it's a decision theory problem and this is why multi-armed bandits are the right framework for A/B testing.
Unfortunately, it takes so much time and effort to understand all the statistical material about testing and power and binomials and whatnot that anything to do with decision theory tends to be relegated to advanced courses or later textbook chapters, if people ever hear of decision theory (or reinforcement learning, or sequential trials, or multi-armed bandits) at all. You might consider decision theory to be the flower of statistics, in which all the cookbooks and algorithms reach their justifications and you know when you can break the various rules of thumb you've learned, but it requires so much time to teach the biology of plants that few students get there or are still interested.
I'm trying to consolidate decision theory materials at https://www.reddit.com/r/DecisionTheory/ https://www.reddit.com/r/DecisionTheory/ (submitted at https://news.ycombinator.com/item?id=10897324 https://news.ycombinator.com/item?id=10897324 ) if you want to read up more on this topic.