3 ms·

To preface this comment, I generally really like Evan Miller's writing on statistics, but this statement: > anyone who lacks a firm understanding of statistica

To preface this comment, I generally really like Evan Miller's writing on statistics, but this statement: > anyone who lacks a firm understanding of statistical power should not be designing or interpreting A/B tests. Completely ignores that you can approach A/B testing from an entirely Bayesian perspective (which Miller has written about fairly well in the past). I might be a bit biased in favor of Bayesian methods, but I'm honestly a bit surprised that many people still run A/B tests using a frequentist frameworks such as this. The biggest reason I would argue to dismiss the NHST (Null-hypothesis significance test) approaching is because A/B Tests are not really controlled experiments, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group. Whether you go NHST or Bayesian is a coin toss in a controlled environment, but when looking at user behavior on a web site it's much better to think in Bayesian terms. Other reasons why a Bayesian frame work should be used: - Marketers care about "Probability A is better than B", not "failure to reject null hypothesis". - If you've been running A/B tests for years, you most certainly have a very good prior over the distribution of conversion rates. It's literally wasting your time and money not to make use of these. - If you're measuring human behavior on a website, there are absolutely confounding variables at play and you want to model these explicitly. It's much easier to do this when thinking in a Bayesian framework. I've been designing and running A/B test of all sorts of complexity at all sorts of companies for a very long time, and my experience has been that approaching the problem from a Bayesian perspective is by far the superior way to run your tests.

ehassler 5y ago
Many A/B tests as RCTs do suffer from the fact that they're randomizations of convenience sample of users coming to a web site, but it's always seemed to me that randomization addresses most confounders due to population heterogeneity. Is there some specific confounding variables you worry about that are not addressed by randomization in this way? One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back. In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?
clircle 5y ago
Even though you can analyze a test Bayesianly, you still need to know about power (or some Bayesian alternative to power) to plan a sample size.
- PheonixPharts 5y ago
  This comment reminded me of yet another reason to prefer the Bayesian set up for A/B tests: you work with the data you have not some bizarre idea of "sample size". Especially with proper priors (which again, everyone running an A/B test should have at this point) you don't have to "plan a sample size". You work with the data you have, and assess the risks in the decision give what you know. There is likewise no risk of "early stopping". If you're doing Bayesian analysis and only have say 100 visitors, and are using priors, the only situation in which you'll have a strong posterior is when the winning variant is notably superior to the current one. All of the ad hoc rules Frequentists put in place are not necessary when you do proper analysis of your posterior probability that A > B. Again we see where Bayesian vs Frequentist doesn't really matter for a clinical trials but does for running an A/B test. In a clinical trial you are going to have to plan how many people are involved anyway, so doing a test power calculation makes sense since you'll need some number and that's a good way to get one. For running an A/B test the bigger problem is usually time not numbers (an exception being email campaigns in which both tend to matter). In the Bayesian setup you if your boss says "I need an answer tomorrow morning" you can take the data you've got, show the probabilities of improvement as well as the distribution of the risks if you're wrong. This allows you to make riskier moves when it makes sense, and be more conservative when it does not. This is something that the out-of-the-box NHST does not allow you.
  - clircle 5y ago
    Well, at least in my day job, a very common question is "how much more data do we need to collect?" or "how long will the experiment take?". A response of "The analysis will be Bayesian, so those questions don't apply" is not helpful. Planning matters! Even a Bayesian with a proper prior can make good guesses about sample sizes, for example, by saying that they have a goal to reduce the length of the 95% credible interval for the most relevant parameter by 80%.
bkfunk 5y ago
I generally agree about the pros you list of Bayesian, but I think in companies that are doing a lot of experiments and are not just optimizing a checkout page, they don't hold as well. For example, you often are testing out features that really are different from any feature before, and so a prior is harder to get alignment on. There are also frequentist methods like CUPED to mitigate the effect of confounding variables, and then usually you're analyzing the results across user segments so that you can try to look for heterogeneous treatment effects. (FWIW, having analyzed a lot of experiments, I've been surprised at how often there is NOT a heterogeneous treatment effect across user segments; the baselines are often very different, but the direction and general magnitude of the lifts are typically similar. And, the confounder that is most often relevant is simply some broad "engagement" dimension: users who use the product a lot might behave differently from users who do not.) In my experience, the biggest blocker to using Bayesian approaches has been about data and computation requirements; there are some closed-form solutions (e.g. https://www.evanmiller.org/bayesian-ab-testing.html https://www.evanmiller.org/bayesian-ab-testing.html , also https://www.chrisstucchio.com/blog/2014/bayesian_asymptotics.html https://www.chrisstucchio.com/blog/2014/bayesian_asymptotics... ) but even those are computationally difficult at scale even if you don't have to do a lot of MCMC sampling. But what is really interesting to me about your comment is: > A/B Tests are not really controlled experiments, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group Could you expand on that? Isn't clinical data also observational: you observe the patient after some time, and see what their symptoms are or what various endpoint measurements are?
- PheonixPharts 5y ago
  > Could you expand on that? Isn't clinical data also observational Generally in statistics there is a divide between controlled experiments and observational studies. The former is how most medical trials are run and the latter is how most anthropologists work. In medical trials you can make sure all the demographics, lifestyle differences etc are controlled for before you even start the trial. In the more extreme case of experiments in the physical sciences you can often tightly control for everything involved where the only difference between test and control is precisely the variable of interest. In anthropology you can't go back in time and say "what if this society had a higher ratio of male/female and didn't go to war?", you can only model what happened and attempt to bake some causal assumptions into your model to see what happened. This is why detailed regression analysis is very important in these fields. Having a test and control group in your A/B test is not really enough to establish the equivalent of a controlled study of rats in a laboratory, or bacteria in petri dishes. In my experience it really helps to include causal models regarding what you know of your A/B testing population to ensure that what you're observing is really the case. I've had concert examples where it looks like one variant is winning and then checking for the causal assumptions find that we were accidentally measuring something else. In practice you can reframe all of this into being some sort of ANOVA test and fit it just fine into a classical framework, but I find starting with a Bayesian methodology make the process easier to reason about.

ehassler 5y ago

Many A/B tests as RCTs do suffer from the fact that they're randomizations of convenience sample of users coming to a web site, but it's always seemed to me that randomization addresses most confounders due to population heterogeneity. Is there some specific confounding variables you worry about that are not addressed by randomization in this way? One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back. In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?

clircle 5y ago

Even though you can analyze a test Bayesianly, you still need to know about power (or some Bayesian alternative to power) to plan a sample size.

PheonixPharts 5y ago
This comment reminded me of yet another reason to prefer the Bayesian set up for A/B tests: you work with the data you have not some bizarre idea of "sample size". Especially with proper priors (which again, everyone running an A/B test should have at this point) you don't have to "plan a sample size". You work with the data you have, and assess the risks in the decision give what you know. There is likewise no risk of "early stopping". If you're doing Bayesian analysis and only have say 100 visitors, and are using priors, the only situation in which you'll have a strong posterior is when the winning variant is notably superior to the current one. All of the ad hoc rules Frequentists put in place are not necessary when you do proper analysis of your posterior probability that A > B. Again we see where Bayesian vs Frequentist doesn't really matter for a clinical trials but does for running an A/B test. In a clinical trial you are going to have to plan how many people are involved anyway, so doing a test power calculation makes sense since you'll need some number and that's a good way to get one. For running an A/B test the bigger problem is usually time not numbers (an exception being email campaigns in which both tend to matter). In the Bayesian setup you if your boss says "I need an answer tomorrow morning" you can take the data you've got, show the probabilities of improvement as well as the distribution of the risks if you're wrong. This allows you to make riskier moves when it makes sense, and be more conservative when it does not. This is something that the out-of-the-box NHST does not allow you.
- clircle 5y ago
  Well, at least in my day job, a very common question is "how much more data do we need to collect?" or "how long will the experiment take?". A response of "The analysis will be Bayesian, so those questions don't apply" is not helpful. Planning matters! Even a Bayesian with a proper prior can make good guesses about sample sizes, for example, by saying that they have a goal to reduce the length of the 95% credible interval for the most relevant parameter by 80%.

bkfunk 5y ago

I generally agree about the pros you list of Bayesian, but I think in companies that are doing a lot of experiments and are not just optimizing a checkout page, they don't hold as well. For example, you often are testing out features that really are different from any feature before, and so a prior is harder to get alignment on. There are also frequentist methods like CUPED to mitigate the effect of confounding variables, and then usually you're analyzing the results across user segments so that you can try to look for heterogeneous treatment effects. (FWIW, having analyzed a lot of experiments, I've been surprised at how often there is NOT a heterogeneous treatment effect across user segments; the baselines are often very different, but the direction and general magnitude of the lifts are typically similar. And, the confounder that is most often relevant is simply some broad "engagement" dimension: users who use the product a lot might behave differently from users who do not.) In my experience, the biggest blocker to using Bayesian approaches has been about data and computation requirements; there are some closed-form solutions (e.g. https://www.evanmiller.org/bayesian-ab-testing.html https://www.evanmiller.org/bayesian-ab-testing.html , also https://www.chrisstucchio.com/blog/2014/bayesian_asymptotics.html https://www.chrisstucchio.com/blog/2014/bayesian_asymptotics... ) but even those are computationally difficult at scale even if you don't have to do a lot of MCMC sampling. But what is really interesting to me about your comment is: > A/B Tests are not really controlled experiments, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group Could you expand on that? Isn't clinical data also observational: you observe the patient after some time, and see what their symptoms are or what various endpoint measurements are?

PheonixPharts 5y ago
> Could you expand on that? Isn't clinical data also observational Generally in statistics there is a divide between controlled experiments and observational studies. The former is how most medical trials are run and the latter is how most anthropologists work. In medical trials you can make sure all the demographics, lifestyle differences etc are controlled for before you even start the trial. In the more extreme case of experiments in the physical sciences you can often tightly control for everything involved where the only difference between test and control is precisely the variable of interest. In anthropology you can't go back in time and say "what if this society had a higher ratio of male/female and didn't go to war?", you can only model what happened and attempt to bake some causal assumptions into your model to see what happened. This is why detailed regression analysis is very important in these fields. Having a test and control group in your A/B test is not really enough to establish the equivalent of a controlled study of rats in a laboratory, or bacteria in petri dishes. In my experience it really helps to include causal models regarding what you know of your A/B testing population to ensure that what you're observing is really the case. I've had concert examples where it looks like one variant is winning and then checking for the causal assumptions find that we were accidentally measuring something else. In practice you can reframe all of this into being some sort of ANOVA test and fit it just fine into a classical framework, but I find starting with a Bayesian methodology make the process easier to reason about.