6 ms·

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to treat 1,000 patients, thinking that the effect would be small. This is one reason I think Bayesian approaches are better than frequentist approaches for A/B testing.

aalleavitch 8y ago
http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- tedsanders 8y ago
  >With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.
  - stochastic_monk 8y ago
    Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
    - tomrod 8y ago
      Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
      - stochastic_monk 8y ago
        A lot of real world cases have imbalanced reward functions. Autonomous vehicles are another example. I don’t doubt regulation systems on spacecraft are similar.
        tomrod 8y ago
        Great examples.
      - webmaven 8y ago
        Consider Facebook's news feed experiment on user's emotions: https://www.nytimes.com/2014/06/30/technology/facebook-tinkers-with-users-emotions-in-news-feed-experiment-stirring-outcry.html https://www.nytimes.com/2014/06/30/technology/facebook-tinke...
    - tedsanders 8y ago
      Even in web testing there isn't an equality between false positives and false negatives. Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests. Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.
  - rpier001 8y ago
    Except that the quote reflects a common misunderstanding. The problem of optimal stopping is mostly a function of decision making over multiple looks. A Bayesian approach that makes decisions over multiple looks has similar issues. This can be mitigated by strong priors, but typically to an unknown degree and at some cost to 'power'. How/why does this claim arise? It is because a 'true' Bayesian approach makes no decisions/inferences - it just describes the current state of knowledge. If describing the distribution of the posterior, then 'stop at any time and be valid' else 'you got your Neyman-Pearson in my Bayesian analysis'.
boron1006 8y ago
Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake. In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size. The cost of the Bayesian approach, in terms of learning a new system of statistics, programming everything up from scratch, and interpreting results, probably wouldn't be worth the efficiency gains. If we were creating an AB testing program, then I would probably do so.
- gbrown 8y ago
  Adaptive designs and early stopping rules are really good when: 1. Data collection is expensive (time or money) 2. Keeping with the status quo in the presence of new evidence is problematic (withholding a promising new drug) 3. Continuing with the experiment in the presence of new evidence is problematic (the new drug is hurting people) Absent one of those features, it's probably not worth the added complexity.
yichijin 8y ago
Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple looks that are planned in advance. At Optimizely we essentially have a continuous version of this that does in fact allow for multiple looks with rigorous control of Type 1 error. As tedsanders mentions, the key upside is that if you start an experiment with a larger-than-expected lift, you can terminate it early. Then over many repeated experiments, you gain a lot in terms of average time to significance. The dissonance in this discussion mostly stems from the fact that this paper (which we actually collaborated on!) uses data from 2014, before we rolled out this new Stats Engine. For more, I would encourage a look at our paper: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
- smallnamespace 8y ago
  What's the tradeoff vs. just taking a direct Bayesian approach? In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem? If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move this widget up the page, change the color, engage with user this way), and a reasonable way to associate those two, then isn't the company literally doing reinforcement learning over time? It seems one benefit of a reinforcement learning framework is it maintains a set of actions that will still be explored in the future without forcing you to prematurely 'choose' whether A or B is actually better—if A is better in reality, then it will be explored more and more often and B will progressively become downweighted over time.
  - dmichulke 8y ago
    > why use an inferential framework at all Probably because of the maintenance cost of the code that was only explored but never exploited.
paulddraper 8y ago
Agreed. The point of most A/B testing is not to find the best approach per se, but have the greatest number of successes. If failures essential don't matter (e.g. number of bacteria killed in a petri dish), sure use frequentist p-value. If failure do matter (e.g. number of patients killed), use Bayesian multi-armed bandit. Wrote a blog post on it: https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-peeking https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-p...
YeGoblynQueenne 8y ago
>> If an effect is strong, it will show up earlier. But if an effect shows up earlier, it is not necessarily strong. I think that's the point with running an experiment for a predeterimned length- so you know you didn't get un/lucky and hit a clump of results at the start of the experiments that will sort of average out later on. Obviously, if a trial drug is killing your patients at a surprising rate, you need to stop the experiment. In fact, I believe experiments are sometimes stopped on ethical grounds when a drugs is found to heal the experiment group at a high rate, also, either so that the control group can also benefit, or just because it is hard to justify giving only half of your patients a life-saving drug and a placebo to the rest. But those are ethical considerations - not practical ones. Such experiments are cut short without complete confidence to the results, when there is the merest hint of ethical issues down the line. At least that's my understanding.
gbrown 8y ago
Adaptive designs are great, but especially in low risk areas like business A/B testing, I'll settle for more people understanding that statistical models are not magic black box truth detectors, and that p-values lose their interpretation in the presence of exploratory practices. The world would probably be a better place if we taught introductory statistics from a Bayesian perspective, but people get pretty set on their ways.

aalleavitch 8y ago

http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html

tedsanders 8y ago
>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.
- stochastic_monk 8y ago
  Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
  - tomrod 8y ago
    Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
    - stochastic_monk 8y ago
      A lot of real world cases have imbalanced reward functions. Autonomous vehicles are another example. I don’t doubt regulation systems on spacecraft are similar.
      - tomrod 8y ago
        Great examples.
    - webmaven 8y ago
      Consider Facebook's news feed experiment on user's emotions: https://www.nytimes.com/2014/06/30/technology/facebook-tinkers-with-users-emotions-in-news-feed-experiment-stirring-outcry.html https://www.nytimes.com/2014/06/30/technology/facebook-tinke...
  - tedsanders 8y ago
    Even in web testing there isn't an equality between false positives and false negatives. Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests. Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.
- rpier001 8y ago
  Except that the quote reflects a common misunderstanding. The problem of optimal stopping is mostly a function of decision making over multiple looks. A Bayesian approach that makes decisions over multiple looks has similar issues. This can be mitigated by strong priors, but typically to an unknown degree and at some cost to 'power'. How/why does this claim arise? It is because a 'true' Bayesian approach makes no decisions/inferences - it just describes the current state of knowledge. If describing the distribution of the posterior, then 'stop at any time and be valid' else 'you got your Neyman-Pearson in my Bayesian analysis'.

boron1006 8y ago

Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake. In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size. The cost of the Bayesian approach, in terms of learning a new system of statistics, programming everything up from scratch, and interpreting results, probably wouldn't be worth the efficiency gains. If we were creating an AB testing program, then I would probably do so.

gbrown 8y ago
Adaptive designs and early stopping rules are really good when: 1. Data collection is expensive (time or money) 2. Keeping with the status quo in the presence of new evidence is problematic (withholding a promising new drug) 3. Continuing with the experiment in the presence of new evidence is problematic (the new drug is hurting people) Absent one of those features, it's probably not worth the added complexity.

yichijin 8y ago

Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple looks that are planned in advance. At Optimizely we essentially have a continuous version of this that does in fact allow for multiple looks with rigorous control of Type 1 error. As tedsanders mentions, the key upside is that if you start an experiment with a larger-than-expected lift, you can terminate it early. Then over many repeated experiments, you gain a lot in terms of average time to significance. The dissonance in this discussion mostly stems from the fact that this paper (which we actually collaborated on!) uses data from 2014, before we rolled out this new Stats Engine. For more, I would encourage a look at our paper: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...

smallnamespace 8y ago
What's the tradeoff vs. just taking a direct Bayesian approach? In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem? If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move this widget up the page, change the color, engage with user this way), and a reasonable way to associate those two, then isn't the company literally doing reinforcement learning over time? It seems one benefit of a reinforcement learning framework is it maintains a set of actions that will still be explored in the future without forcing you to prematurely 'choose' whether A or B is actually better—if A is better in reality, then it will be explored more and more often and B will progressively become downweighted over time.
- dmichulke 8y ago
  > why use an inferential framework at all Probably because of the maintenance cost of the code that was only explored but never exploited.

paulddraper 8y ago

Agreed. The point of most A/B testing is not to find the best approach per se, but have the greatest number of successes. If failures essential don't matter (e.g. number of bacteria killed in a petri dish), sure use frequentist p-value. If failure do matter (e.g. number of patients killed), use Bayesian multi-armed bandit. Wrote a blog post on it: https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-peeking https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-p...

YeGoblynQueenne 8y ago

>> If an effect is strong, it will show up earlier. But if an effect shows up earlier, it is not necessarily strong. I think that's the point with running an experiment for a predeterimned length- so you know you didn't get un/lucky and hit a clump of results at the start of the experiments that will sort of average out later on. Obviously, if a trial drug is killing your patients at a surprising rate, you need to stop the experiment. In fact, I believe experiments are sometimes stopped on ethical grounds when a drugs is found to heal the experiment group at a high rate, also, either so that the control group can also benefit, or just because it is hard to justify giving only half of your patients a life-saving drug and a placebo to the rest. But those are ethical considerations - not practical ones. Such experiments are cut short without complete confidence to the results, when there is the merest hint of ethical issues down the line. At least that's my understanding.

gbrown 8y ago

Adaptive designs are great, but especially in low risk areas like business A/B testing, I'll settle for more people understanding that statistical models are not magic black box truth detectors, and that p-values lose their interpretation in the presence of exploratory practices. The world would probably be a better place if we taught introductory statistics from a Bayesian perspective, but people get pretty set on their ways.