22 ms·

Most Winning A/B Test Results Are Illusory [pdf]

godDLL 10y ago
This is very well explained, even if you don't understand statistics. Apparently not many vendors of A/B testing software do.
- iaw 10y ago
  I suspect the people building it do, the people selling it probably do not.
  - CalRobert 10y ago
    Having worked between customers and engineering at a place that did this, I can confirm that "our tests are Bayesian!!" was a refrain everyone was taught to repeat, but few if any were taught what it meant. This video is helpful: https://www.youtube.com/watch?v=Dy_LRK2Pkig https://www.youtube.com/watch?v=Dy_LRK2Pkig I still don't know a great way to describe it in the 6 or 7 seconds you have before the potential customer's attention starts to flag.
    - iaw 10y ago
      > I still don't know a great way to describe it in the 6 or 7 seconds you have before the potential customer's attention starts to flag. I don't think there is a 6-7 second way to describe it. I mean, imagine someone that knows nothing of frequentist statistics, could you explain the meaning/significance of a confidence interval to them? It's a pretty high bar to clear, I've found for CI's it's typically 1-5 minutes depending on background.
tedsanders 10y ago
I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testing.) The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate? The wrong question to ask is: Is A different/better than B, given some confidence threshold? The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold. A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking how big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking is there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value. Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.
- jgalt212 10y ago
  > The wrong question to ask is: Is A better than B, given some confidence threshold? That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.
  - tedsanders 10y ago
    Suppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences. It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
    - imh 10y ago
      In A/B testing, either A or B is the control.
- martingoodson 10y ago
  I don't know about 'flawed'. Classical hypothesis testing has its place, when used correctly. I guess the point of the article is that it falls apart dramatically when you do certain things wrong. And that these things turn out to be in very common practice. Disclaimer: I wrote that article (a long time ago...)
- Obi_Juan_Kenobi 10y ago
  That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not. As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?
Vinnl 10y ago
Nice article. One question though: > Perform a second ‘validation’ test repeating your original test to check that the effect is real Isn't this just the same as taking a larger sample size?
- Obi_Juan_Kenobi 10y ago
  Almost. The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usually your assumption is that these are comparable, but that's not necessarily true. Furthermore, the effect is only meaningful if that assumption is true, for most business cases (i.e. you want this effect to hold in the future). In practice, a lot of effects do disappear when you repeat a test because there was some unaccounted for factor that varied between them. It's a good sanity check. But you're right, much of the purpose is to discover mean regression. This is something that happens more often than you'd expect because you tend to be focusing large effects, many of which will simply be due to chance.
  - Vinnl 10y ago
    So basically, it means waiting a while before performing the test again? Since "larger sample size" mostly just means "keep the test running for longer", so the difference is that in that case, the "second" test is very close in time to the "first" test?
    - EdHominem 10y ago
      Not entirely. Even if run right after one another, running two tests buckets the results into early/late, which simply using a larger sample set does not. Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
      - Vinnl 10y ago
        So what would the conclusions be in the two-test and large-test case? My assumption would be that if one of the two tests would fail, then the correlation in a large test would also have been so low as to mean no effect?
jkuria 10y ago
Hmmh, this is interesting. Most A/B software will let you set a level of statistical confidence that needs to be attained before a winner can be declared. For example in Google Analytics two common ones are 95% and 99%. We stop our tests when they reach at least 95% confidence. Is the author saying one must wait for 6000 events even if the difference between A/B is large? The larger the relative difference, the fewer events needed.
- squarecog 10y ago
  Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion. You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).
  - tedsanders 10y ago
    Is it really a failure mode? I think it should be fine to stop whenever you want during a test. You just have to be smart enough to not use a t test and misinterpret it to mean than A is 95% likelier than B.
    - daveguy 10y ago
      Yes. It really is a failure mode. No, it is not fine to stop whenever you want during a test. It doesn't matter what statistical test you are using -- you need to gather enough data to have sufficient statistical power. Also, the more tests you do the lower the p-value should be. If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
      - 10y ago
ted_dunning 10y ago
This is yet another article that ignores the fact that there is a MUCH better approach to this problem. Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more "Which alternative is nearly as good as the best with pretty high probability?". This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem. The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement. See http://tdunning.blogspot.com/2012/02/bayesian-bandits.html http://tdunning.blogspot.com/2012/02/bayesian-bandits.html for an essay and see https://github.com/tdunning/bandit-ranking https://github.com/tdunning/bandit-ranking for an example applied to ranking.
- mattj 10y ago
  I agree with you (and love your blog, btw), but I think you're skipping over at least a few benefits you can get out of a mature / well built a/b framework that are hard to build into a bandit approach. The biggest one I've found personally useful is days-in analysis; for example, quantifying the impact of a signup-time experiment on one-week retention. This doesn't really apply to learning ranking functions or other transactional (short-feedback loop) optimization. That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.
  - paulddraper 10y ago
    Speaking of mature, well-built A/B test frameworks, Google Analytics uses multi-armed bandit. https://support.google.com/analytics/answer/2844870?hl=en https://support.google.com/analytics/answer/2844870?hl=en
RA_Fisher 10y ago
Any other statisticians want to champion this? https://news.ycombinator.com/item?id=13434410 https://news.ycombinator.com/item?id=13434410
ssharp 10y ago
For most businesses, focusing on the math and making incremental improvements to the statistical methodology is a waste of time. There is a "good enough" approach to using tools like Optimizely and VWO. Instead, they should be focusing on the quality of the tests they run. Quality hypotheses, preferably backed by data-driven insights into behavior that test very clear, if not dramatic, changes to the user flow will be what leads to bottom-line improvements, not increasing the mathematical merit and rigor of poorly-conceived tests. Of course, doing both is ideal but I put more fault on the testing software than the companies using it. Keep track of historical conversion rate and adjust/account for noise in your historical conversion rate. Conceive a testing program that focuses on the quality testing and you'll likely see an upward trend in that historical conversion rate.
guico 10y ago
I don't get this: "But remember: if you limit yourself to detecting uplifts of over 10% you will miss also miss negative effects of less than 10%. Can you afford to reduce your conversion rate by 5% without realizing it?" You'd potentially lose 5% CR if you ship the variant even when it doesn't show a detectable uplift. Why would you do that?
youngtaff 10y ago
Pity they didn't add a date to the paper!

This is very well explained, even if you don't understand statistics. Apparently not many vendors of A/B testing software do.

iaw 10y ago
I suspect the people building it do, the people selling it probably do not.
- CalRobert 10y ago
  Having worked between customers and engineering at a place that did this, I can confirm that "our tests are Bayesian!!" was a refrain everyone was taught to repeat, but few if any were taught what it meant. This video is helpful: https://www.youtube.com/watch?v=Dy_LRK2Pkig https://www.youtube.com/watch?v=Dy_LRK2Pkig I still don't know a great way to describe it in the 6 or 7 seconds you have before the potential customer's attention starts to flag.
  - iaw 10y ago
    > I still don't know a great way to describe it in the 6 or 7 seconds you have before the potential customer's attention starts to flag. I don't think there is a 6-7 second way to describe it. I mean, imagine someone that knows nothing of frequentist statistics, could you explain the meaning/significance of a confidence interval to them? It's a pretty high bar to clear, I've found for CI's it's typically 1-5 minutes depending on background.

tedsanders 10y ago

I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testing.) The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate? The wrong question to ask is: Is A different/better than B, given some confidence threshold? The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold. A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking how big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking is there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value. Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.

jgalt212 10y ago
> The wrong question to ask is: Is A better than B, given some confidence threshold? That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.
- tedsanders 10y ago
  Suppose A is significantly better than the control. And B is not significantly better than the control. Does that imply A is significantly better than B? No. Statistical differences do not the follow the same logic as normal differences. It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
  - imh 10y ago
    In A/B testing, either A or B is the control.
martingoodson 10y ago
I don't know about 'flawed'. Classical hypothesis testing has its place, when used correctly. I guess the point of the article is that it falls apart dramatically when you do certain things wrong. And that these things turn out to be in very common practice. Disclaimer: I wrote that article (a long time ago...)
Obi_Juan_Kenobi 10y ago
That's putting the cart before the horse. Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not. As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?

Vinnl 10y ago

Nice article. One question though: > Perform a second ‘validation’ test repeating your original test to check that the effect is real Isn't this just the same as taking a larger sample size?

Obi_Juan_Kenobi 10y ago
Almost. The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usually your assumption is that these are comparable, but that's not necessarily true. Furthermore, the effect is only meaningful if that assumption is true, for most business cases (i.e. you want this effect to hold in the future). In practice, a lot of effects do disappear when you repeat a test because there was some unaccounted for factor that varied between them. It's a good sanity check. But you're right, much of the purpose is to discover mean regression. This is something that happens more often than you'd expect because you tend to be focusing large effects, many of which will simply be due to chance.
- Vinnl 10y ago
  So basically, it means waiting a while before performing the test again? Since "larger sample size" mostly just means "keep the test running for longer", so the difference is that in that case, the "second" test is very close in time to the "first" test?
  - EdHominem 10y ago
    Not entirely. Even if run right after one another, running two tests buckets the results into early/late, which simply using a larger sample set does not. Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
    - Vinnl 10y ago
      So what would the conclusions be in the two-test and large-test case? My assumption would be that if one of the two tests would fail, then the correlation in a large test would also have been so low as to mean no effect?

jkuria 10y ago

Hmmh, this is interesting. Most A/B software will let you set a level of statistical confidence that needs to be attained before a winner can be declared. For example in Google Analytics two common ones are 95% and 99%. We stop our tests when they reach at least 95% confidence. Is the author saying one must wait for 6000 events even if the difference between A/B is large? The larger the relative difference, the fewer events needed.

squarecog 10y ago
Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion. You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).
- tedsanders 10y ago
  Is it really a failure mode? I think it should be fine to stop whenever you want during a test. You just have to be smart enough to not use a t test and misinterpret it to mean than A is 95% likelier than B.
  - daveguy 10y ago
    Yes. It really is a failure mode. No, it is not fine to stop whenever you want during a test. It doesn't matter what statistical test you are using -- you need to gather enough data to have sufficient statistical power. Also, the more tests you do the lower the p-value should be. If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
    - 10y ago

ted_dunning 10y ago

This is yet another article that ignores the fact that there is a MUCH better approach to this problem. Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more "Which alternative is nearly as good as the best with pretty high probability?". This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem. The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement. See http://tdunning.blogspot.com/2012/02/bayesian-bandits.html http://tdunning.blogspot.com/2012/02/bayesian-bandits.html for an essay and see https://github.com/tdunning/bandit-ranking https://github.com/tdunning/bandit-ranking for an example applied to ranking.

mattj 10y ago
I agree with you (and love your blog, btw), but I think you're skipping over at least a few benefits you can get out of a mature / well built a/b framework that are hard to build into a bandit approach. The biggest one I've found personally useful is days-in analysis; for example, quantifying the impact of a signup-time experiment on one-week retention. This doesn't really apply to learning ranking functions or other transactional (short-feedback loop) optimization. That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.
- paulddraper 10y ago
  Speaking of mature, well-built A/B test frameworks, Google Analytics uses multi-armed bandit. https://support.google.com/analytics/answer/2844870?hl=en https://support.google.com/analytics/answer/2844870?hl=en

RA_Fisher 10y ago

Any other statisticians want to champion this? https://news.ycombinator.com/item?id=13434410 https://news.ycombinator.com/item?id=13434410

ssharp 10y ago

For most businesses, focusing on the math and making incremental improvements to the statistical methodology is a waste of time. There is a "good enough" approach to using tools like Optimizely and VWO. Instead, they should be focusing on the quality of the tests they run. Quality hypotheses, preferably backed by data-driven insights into behavior that test very clear, if not dramatic, changes to the user flow will be what leads to bottom-line improvements, not increasing the mathematical merit and rigor of poorly-conceived tests. Of course, doing both is ideal but I put more fault on the testing software than the companies using it. Keep track of historical conversion rate and adjust/account for noise in your historical conversion rate. Conceive a testing program that focuses on the quality testing and you'll likely see an upward trend in that historical conversion rate.

guico 10y ago

I don't get this: "But remember: if you limit yourself to detecting uplifts of over 10% you will miss also miss negative effects of less than 10%. Can you afford to reduce your conversion rate by 5% without realizing it?" You'd potentially lose 5% CR if you ship the variant even when it doesn't show a detectable uplift. Why would you do that?

youngtaff 10y ago

Pity they didn't add a date to the paper!