5 ms·

A Better Approach to A/B Test Analysis

51 points by bvanvugt 12y ago

jessicaraygun 12y ago
Hey! I'm the author of Confidence.js. Emily Malcolm and I have been working hard on this new approach for the past few weeks and we're super excited to share it! We're both here to answer any questions :)
- kiyoto 12y ago
  This is really cool. I can tell you guys actually know statistics. w/r/t testing multiple hypotheses, I've long wondered why false discovery rate never gets mentioned in the A/B-testing context. Any thoughts?
  - jessicaraygun 12y ago
    Thanks for your question! I'll pass it on to Emily, our stats expert.
- tedsanders 12y ago
  >First, we use Chi Squared Tests to determine if differences in the A/B test data are meaningful or not. Could you explain why you've chosen to take a binary approach to determining whether differences are meaningful or not? To me it seems like a continuous approach would be both more useful and more realistic. Creating an artificial threshold for significance seems a bit silly (and it also makes the model harder for users to use, because different applications might need different significance levels to justify an action). From my perspective, every data point contains information and if you wait for significance you're essentially ignoring early information. Edit: Also, when switching costs are small, significance levels become mostly pointless and you just want to switch to the best A/B option immediately. As evidence swings the other way, you just switch back.
  - em441 12y ago
    Hi! This is Emily, the stats brains behind this. My research indicated that 80% significance was commonly used for A/B Testing (and of course any significance level >80% would be even more conservative). I definitely see your point of the advantage of a continuous test and I think for the expert being able to see the exact significance level for the test would be very useful. However, our thoughts were that the average user might not necessarily have the expertise to know what a "good enough" level of significance would be for any test. Rather than having to educate every user on what significance means and how to interpret it we decided that a "yes, significant" or "no, not significant" would be more easily interpreted by all, regardless of their statistical background. If there is demand for a more continuous approach, it certainly could easily be implemented.
    - tedsanders 12y ago
      Thanks for the reply. I guess it's a good reminder for everyone building statistical products (or even non-statistical products) that the goal is not an optimal formula, but a result that makes users comfortable.
adamcowley 12y ago
Nice to see someone questioning old statistical methods and then bringing new methods to the masses.
wtracy 12y ago
The statistical methods we use now were created in the context of long-running experiments that had to be set up in advance and run in parallel. In that situation, you have to decide up-front how many subjects to test on, and the methods reflect this. I'd like to see someone takle creating a method aimed at our situations where results steadily trickle in. There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"
- jessicaraygun 12y ago
  Yes! This is very important to us in dealing with transactional email. Our Z-Test Method in Confidence.js pinpoints when results become statistically significant by calculating the required sample size at any given time using the standard error, zScore, and margin of error. If we have more samples than required, we can stop our test with confidence that we have statistically significant results. We're still working out a way to do this with the new method - we have a hunch it might have to do with identifying when the confidence level becomes stable (as it does with more samples over time).
- em441 12y ago
  Hi! This is Emily, the stats brains behind this. I think this is a really interesting and important question. I know this question comes up in other contexts (clinical trials) and there has been a lot of work done on how to find "stopping times" from a statistical standpoint. I'd love to see some of these statistical methods for sequential analysis (developed for clinical trials) integrated into how we do A/B testing.
- tedsanders 12y ago
  I totally agree. And I think Bayesian methods (which compute the likelihood of the model given the data) tend to work much better for these rolling data applications than frequentist methods (which compute the likelihood of the data given the model). The problem for frequentist methods here is that when data is constantly rolling in it's hard to specify the space of all possible data collected (because traditional bounds like the number of data points no longer work). Here's a link to a paper on a Bayesian approach to the multi-armed bandit problem: http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract;jsessionid=364E386ABCD27DA0D592DC7810CA558A.f01t03 http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract...
mrmch 12y ago
Worth pointing out that our (sendwithus) js library for calculating all of this is open source: github.com/sendwithus/confidence
vincentbarr 12y ago
There are a number of testing tools relying on this methodology. This is useful: Evan's Awesome A/B Tools (http://www.evanmiller.org/ab-testing/ http://www.evanmiller.org/ab-testing/). It includes a Chi-Squared test, sample size calculator, two sample T-test, and Poisson Means test.
robdoherty2 12y ago
At my company we've implemented a Bayesian A/B test in order to minimize the amount of time a test has to run. http://visualrevenue.com/blog/2013/02/tech-bayesian-instant-headline-testing.html http://visualrevenue.com/blog/2013/02/tech-bayesian-instant-...
- em441 12y ago
  I will definitely have a look through this. Thanks!
_deh 12y ago
Here's another paper that's relevant to this topic: http://www.qubitproducts.com/sites/default/files/pdf/most_winning_ab_test_results_are_illusory.pdf http://www.qubitproducts.com/sites/default/files/pdf/most_wi.... And discussion here: https://news.ycombinator.com/item?id=7287665 https://news.ycombinator.com/item?id=7287665

51 points by bvanvugt 12y ago

Hey! I'm the author of Confidence.js. Emily Malcolm and I have been working hard on this new approach for the past few weeks and we're super excited to share it! We're both here to answer any questions :)

kiyoto 12y ago
This is really cool. I can tell you guys actually know statistics. w/r/t testing multiple hypotheses, I've long wondered why false discovery rate never gets mentioned in the A/B-testing context. Any thoughts?
- jessicaraygun 12y ago
  Thanks for your question! I'll pass it on to Emily, our stats expert.
tedsanders 12y ago
>First, we use Chi Squared Tests to determine if differences in the A/B test data are meaningful or not. Could you explain why you've chosen to take a binary approach to determining whether differences are meaningful or not? To me it seems like a continuous approach would be both more useful and more realistic. Creating an artificial threshold for significance seems a bit silly (and it also makes the model harder for users to use, because different applications might need different significance levels to justify an action). From my perspective, every data point contains information and if you wait for significance you're essentially ignoring early information. Edit: Also, when switching costs are small, significance levels become mostly pointless and you just want to switch to the best A/B option immediately. As evidence swings the other way, you just switch back.
- em441 12y ago
  Hi! This is Emily, the stats brains behind this. My research indicated that 80% significance was commonly used for A/B Testing (and of course any significance level >80% would be even more conservative). I definitely see your point of the advantage of a continuous test and I think for the expert being able to see the exact significance level for the test would be very useful. However, our thoughts were that the average user might not necessarily have the expertise to know what a "good enough" level of significance would be for any test. Rather than having to educate every user on what significance means and how to interpret it we decided that a "yes, significant" or "no, not significant" would be more easily interpreted by all, regardless of their statistical background. If there is demand for a more continuous approach, it certainly could easily be implemented.
  - tedsanders 12y ago
    Thanks for the reply. I guess it's a good reminder for everyone building statistical products (or even non-statistical products) that the goal is not an optimal formula, but a result that makes users comfortable.

adamcowley 12y ago

Nice to see someone questioning old statistical methods and then bringing new methods to the masses.

wtracy 12y ago

The statistical methods we use now were created in the context of long-running experiments that had to be set up in advance and run in parallel. In that situation, you have to decide up-front how many subjects to test on, and the methods reflect this. I'd like to see someone takle creating a method aimed at our situations where results steadily trickle in. There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"

jessicaraygun 12y ago
Yes! This is very important to us in dealing with transactional email. Our Z-Test Method in Confidence.js pinpoints when results become statistically significant by calculating the required sample size at any given time using the standard error, zScore, and margin of error. If we have more samples than required, we can stop our test with confidence that we have statistically significant results. We're still working out a way to do this with the new method - we have a hunch it might have to do with identifying when the confidence level becomes stable (as it does with more samples over time).
em441 12y ago
Hi! This is Emily, the stats brains behind this. I think this is a really interesting and important question. I know this question comes up in other contexts (clinical trials) and there has been a lot of work done on how to find "stopping times" from a statistical standpoint. I'd love to see some of these statistical methods for sequential analysis (developed for clinical trials) integrated into how we do A/B testing.
tedsanders 12y ago
I totally agree. And I think Bayesian methods (which compute the likelihood of the model given the data) tend to work much better for these rolling data applications than frequentist methods (which compute the likelihood of the data given the model). The problem for frequentist methods here is that when data is constantly rolling in it's hard to specify the space of all possible data collected (because traditional bounds like the number of data points no longer work). Here's a link to a paper on a Bayesian approach to the multi-armed bandit problem: http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract;jsessionid=364E386ABCD27DA0D592DC7810CA558A.f01t03 http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract...

mrmch 12y ago

Worth pointing out that our (sendwithus) js library for calculating all of this is open source: github.com/sendwithus/confidence

vincentbarr 12y ago

There are a number of testing tools relying on this methodology. This is useful: Evan's Awesome A/B Tools (http://www.evanmiller.org/ab-testing/ http://www.evanmiller.org/ab-testing/). It includes a Chi-Squared test, sample size calculator, two sample T-test, and Poisson Means test.

robdoherty2 12y ago

At my company we've implemented a Bayesian A/B test in order to minimize the amount of time a test has to run. http://visualrevenue.com/blog/2013/02/tech-bayesian-instant-headline-testing.html http://visualrevenue.com/blog/2013/02/tech-bayesian-instant-...

em441 12y ago
I will definitely have a look through this. Thanks!

_deh 12y ago

Here's another paper that's relevant to this topic: http://www.qubitproducts.com/sites/default/files/pdf/most_winning_ab_test_results_are_illusory.pdf http://www.qubitproducts.com/sites/default/files/pdf/most_wi.... And discussion here: https://news.ycombinator.com/item?id=7287665 https://news.ycombinator.com/item?id=7287665