2 ms·
Note, I expect this to be the first in a series. It probably won't wind up laid out exactly as I expect, but here is my tentative plan: 1. Rigorous (this one)
by btilly 14y ago
Note, I expect this to be the first in a series. It probably won't wind up laid out exactly as I expect, but here is my tentative plan:
1. Rigorous (this one) - How to use frequentist techniques to define a very rigorous statistical procedure that - no matter how small the bias in the test - can give correct answers with very high probability (and, of course, support multiple looks at your statistics).
2. Fixed data - Your business has an upper limit on how much data it realistically can collect. How to design a statistical procedure that makes reasonably good decisions while remaining within that limit.
3. Simple test - How to use Bayes' Theorem to design very simple and straightforward test criteria, where every A/B test has a known cost up front.
4. Bayesian statistics - How a Bayesian approach compares to the previous ones. Where the sticking points are, what its advantages are.
5. Bayesian optimization - How a Bayesian approach can let you turn the problem of A/B testing into a pure optimization problem where your decision to stop is based on whether the cost of continuing the test exceeds your expected return from continuing it over some time horizon.
We'll see how far I get. As it goes on, they become tricker to write at the level that I'm aiming for. And it may be that people will be interested in something else. But I thought that the first two were important. If for no other reason than to give me something definitive to point people at who read Evan Miller's article and then were telling people to run A/B tests in a way that made no business sense.
- loup-vaillant 14y agoFrom your article: > We will follow Evan's lead and use a frequentist approach. I though: "how about using the correct approach instead?", followed by a helpful comment about how you are a Bad Person¹. Hem. Then I noticed that the way you talk about p-value doesn't sound very frequenty. And then your comment. I eagerly await for the rest of the sequence –err, series². [1]: https://xkcd.com/386/ https://xkcd.com/386/ [2]: Come to think of it, this does looks like LessWrong material to me.
- jessaustin 14y agoI'm very encouraged by the provided prospectus, since it implies btilly is one of the chosen few who not only know both frequentist and Bayesian techniques but also know when to use each and when not to e.g. give a Bayesian interpretation to a frequentist technique.
- btilly 14y agoThanks for the kind words, but I'm winging this more than I'd like to admit. (Though winging it as carefully as I can.) But if I finish, I'll probably have learned a lot about when to use each. Hopefully because I figured it out correctly, and not because I said something egregiously wrong and got corrected by someone who knew better!
- btilly 14y agoI suspect that you may like me substantially less when I get to discussing the trade-offs that are inherent in Bayesian statistics. I know how wonderful it can be to feel that you have realized The Truth, but the frequentist school of statistics does not remain in existence just because statisticians are too stupid to recognize the obvious superiority of Bayesian methods.
- loup-vaillant 14y agoHaving read chapter 2 of Probability Theory: the Logic of Science by E.T. Jaynes, my probability for the stupidity hypothesis went way, way up. There's probably a heavy status-quo bias at work too. I mean, many frequentist methods directly contradict Cox theorems! That should convince anyone they belong to History, not Science, doesn't it?
- loup-vaillant 14y agoI know the above comment sounds extremist, but bear in mind that its truth value has nothing to do with that. I mean for instance, the only reason "2 + 2 = 4, and anyone who believe otherwise lacks either a brain or some crucial information" does not sound extremist is because everyone actually agrees with it. So. Who in her right mind would reject the three assumptions behind Cox's theorem, and for what reason? Assuming we don't reject those axioms, why should we not reject Frequentism at once? https://en.wikipedia.org/wiki/Cox%27s_theorem https://en.wikipedia.org/wiki/Cox%27s_theorem
- _dps 14y ago> Then I noticed that the way you talk about p-value doesn't sound very frequenty. I think a lot of people improperly understand the Frequentist/Bayesian divide; both look at likelihood functions but the Frequentists assert, as a matter of experimental interpretation, that the likelihood function should depend only on the observed data while the Bayesians are willing to admit "out-of-experiment" modifications to the likelihoods implied by the data (i.e. the "prior"). Frequentist arguments correctly point out that the prior is, from the isolated standpoint of the experimental observations, purely subjective and non-falsifiable. Bayesian arguments correctly point out that experiments are designed by humans in context, and that the context may imply a prior distribution that is not evident within the observations themselves. Neither is incorrect, but they answer different questions. Which question is the right one to answer is context-dependent. As a concrete example, consider an A/B test where A and B perform very similarly. Merely seeing A's performance against B does not carry the contextual information that A has consecutively beaten one hundred previous non-B competitors that sort-of-look-like B. The Frequentist will correctly argue that, if you don't explicitly model the A/B test generation process (as opposed to just the A/B test itself), then all the data can tell you based purely on the laws of probability is that A and B are likely to be pretty similar. The trouble faced by the frequentist is that it's pretty easy to model an individual A/B test for, say, binary outcomes, but it's very hard to explicitly model the generation of A/B test competitors. So the Frequentist says "all I can tell you and still be strictly objective is that A and B look similar". The Bayesian, rather than explicitly modeling the generation of tests as the Frequentist would like, wraps up that contextual knowledge into a subjective prior distribution that heavily weights A over B simply because A has won so many times before. The Bayesian's advantage is that the previously-blocking part of the probability modeling problem is punted into a subjective, experimenter-designed, input. This is a tradeoff, not an increase of correctness! It will "do the right thing" assuming the new B looks a lot like the old Bs, but will do the wrong thing if the new B is dramatically different and maybe looks a lot more like A with small tweaks. Bayesianism is not a universal solution to probability modeling problems, and Frequentism is not an obsolete unhip method.
- darkxanthos 14y agoThank you for adding an objective comment about this whole controversy. I love the ideas present in a Bayesian approach but there seems to be a lot of value in Frequentist statistics as well. I'm glad to see I'm not crazy for thinking the two can find a balanced existence.
- pjungwir 14y ago> And it may be that people will be interested in something else. Personally I'm eagerly awaiting #5: treating the question of whether to continue a test as an optimization problem.
- tel 14y agoI'm excited! I've been looking forward to this for a while from your comments. This first post is written at a great level and answers a very practical question that few people realize even has answers to. I'll be waiting eagerly to read the rest. Cheers!
- gburt 14y agoThank you for this. You are awesome.
- mwexler 14y agoWill 5.1 or 6 discuss some aspects of the Multi Armed Bandit approach, ala http://analytics.blogspot.com/2013/01/multi-armed-bandit-experiments.html http://analytics.blogspot.com/2013/01/multi-armed-bandit-exp... ? Looking forward to the whole series!
- btilly 14y agoI was not planning to go into MAB approaches because that quickly gets into material that I don't know how to easily explain at the level that I am trying to reach here. And if I did go into that material, it would be an extremely idiosyncratic take. But if you're interested in an overview of my general thoughts on MAB approaches, http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algorithms-its.html http://bentilly.blogspot.com/2012/09/ab-testing-vs-mab-algor... may be of interest.
- woah 14y agoI'm a designer, and I value AB tests, but the math required to make sure I can trust my results is not what I want to focus on. There should be a startup that offers a some sort of framework, instruction, and automation for this stuff. I would use it.
- btilly 14y agoThere are a number of startups that do exactly that. However there are some complex trade-offs that need to be made under the hood, and businesses whose goal is to make things simple, naturally tend to shy away from educating their customers on those issues.