3 ms·
A/B testing is a Bayesian decision problem. Hypothesis testing is really the wrong way to frame this problem. Dynamically it is a bandit problem and staticall
by equark 16y ago
A/B testing is a Bayesian decision problem. Hypothesis testing is really the wrong way to frame this problem. Dynamically it is a bandit problem and statically it's just about maximizing expected utility, where the expectation is taken with respect to the posterior distribution. This becomes especially crucial when doing lots of tests or experimenting sequentially. All the frequentist proposals don't have their advertised coverage rates in these situations. Unfortunately most write-ups seem to be completely ignorant of Bayesian statistics.
- btilly 16y agoI mostly agree with you. However if you've ever tried to explain those concepts to a bunch of PMs, you'll quickly discover that it isn't worth the complication of trying to describe things the "right" way. There are more important problems to deal with. My one point of disagreement is that dynamically it is a bandit problem. The reason that I disagree is that the things we are tracking and trying to optimize have very strong time dependencies that aren't obvious. (Behavior varies a lot by time of day, day of week, week of month, and month of year.) Therefore attempting to dynamically optimize can very, very easily go astray. That's not to say you can't do it. For instance I am quite supportive of using dynamic techniques to explore a multi-variate space. But in the end I wouldn't be comfortable unless you did a classic A/B test to validate the results.