6 ms·

Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I

by nanis 8y ago

Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a thing.

p10_user 8y ago
It’s done in academia too. Along with “remove the outlier data that messes up our p values”
- glup 8y ago
  The difference is that the people in academia know that it is statistically unsound and choose to act unethically. I think the problem in A/B testing is that a lot of developers don't know it is unsound.
  - carlmr 8y ago
    This is one of the cases where self-taught developers are usually not as good as those with a "proper" education. In uni you'll learn a lot of tangentially related stuff like p-hacking and design of experiment which a lot of people won't pick up when self-taught.
- whyever 8y ago
  It might make sense to remove systematic outliers if you know they are from non-statistical effects.
IshKebab 8y ago
It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.
- nanis 8y ago
  People use classical methods because it is easier for them to understand than Bayesian. When using classical methods, the least one can do is to fix sample size before the experiment, and not peek until the experiment is over. That is easier than explaining Bayesian methods to people who cannot handle classical Stats.
yichijin 8y ago
Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in clinical trials explicitly allow for multiple looks while maintaining correct control of the Type 1 error rate. At Optimizely we essentially have a version of this where the monitoring can be conducted contiuously with rigorous control of Type 1 error. Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
- Jabbles 8y ago
  Presumably using your method takes longer/requires more samples than a method that only checks once?
  - srean 8y ago
    I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.
    - computerphage 8y ago
      That's highly counter-intuitive to me. Can you try to motivate why that's the case? My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better, at best it could be the same.
      - srean 8y ago
        This is 1940s stuff. Checkout Wald. Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness (). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to that commitment. () If your method ensures correctness even after inspection at intermediate stages then its a sequential method by definition. There is some confusion in literature about Bayesian and sequential. They are orthogonal concepts. Both Bayesian and Frequentist test of hypothesis can be sequential
beagle3 8y ago
> The answer was "until we get a significant result." Done properly, it might be way more efficient than setting your parameters ahead of time, see [0] ; If I had gotten that response from him, I'd assume that's what he meant. [0] https://en.wikipedia.org/wiki/Sequential_probability_ratio_test https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...

by nanis 8y ago

p10_user 8y ago

It’s done in academia too. Along with “remove the outlier data that messes up our p values”

glup 8y ago
The difference is that the people in academia know that it is statistically unsound and choose to act unethically. I think the problem in A/B testing is that a lot of developers don't know it is unsound.
- carlmr 8y ago
  This is one of the cases where self-taught developers are usually not as good as those with a "proper" education. In uni you'll learn a lot of tangentially related stuff like p-hacking and design of experiment which a lot of people won't pick up when self-taught.
whyever 8y ago
It might make sense to remove systematic outliers if you know they are from non-statistical effects.

IshKebab 8y ago

It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.

nanis 8y ago
People use classical methods because it is easier for them to understand than Bayesian. When using classical methods, the least one can do is to fix sample size before the experiment, and not peek until the experiment is over. That is easier than explaining Bayesian methods to people who cannot handle classical Stats.

yichijin 8y ago

Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in clinical trials explicitly allow for multiple looks while maintaining correct control of the Type 1 error rate. At Optimizely we essentially have a version of this where the monitoring can be conducted contiuously with rigorous control of Type 1 error. Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...

Jabbles 8y ago
Presumably using your method takes longer/requires more samples than a method that only checks once?
- srean 8y ago
  I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.
  - computerphage 8y ago
    That's highly counter-intuitive to me. Can you try to motivate why that's the case? My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better, at best it could be the same.
    - srean 8y ago
      This is 1940s stuff. Checkout Wald. Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness (). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to that commitment. () If your method ensures correctness even after inspection at intermediate stages then its a sequential method by definition. There is some confusion in literature about Bayesian and sequential. They are orthogonal concepts. Both Bayesian and Frequentist test of hypothesis can be sequential

beagle3 8y ago

> The answer was "until we get a significant result." Done properly, it might be way more efficient than setting your parameters ahead of time, see [0] ; If I had gotten that response from him, I'd assume that's what he meant. [0] https://en.wikipedia.org/wiki/Sequential_probability_ratio_test https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...