5 ms·

The statistical methods we use now were created in the context of long-running experiments that had to be set up in advance and run in parallel. In that situati

by wtracy 12y ago

The statistical methods we use now were created in the context of long-running experiments that had to be set up in advance and run in parallel. In that situation, you have to decide up-front how many subjects to test on, and the methods reflect this. I'd like to see someone takle creating a method aimed at our situations where results steadily trickle in. There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?"

jessicaraygun 12y ago
Yes! This is very important to us in dealing with transactional email. Our Z-Test Method in Confidence.js pinpoints when results become statistically significant by calculating the required sample size at any given time using the standard error, zScore, and margin of error. If we have more samples than required, we can stop our test with confidence that we have statistically significant results. We're still working out a way to do this with the new method - we have a hunch it might have to do with identifying when the confidence level becomes stable (as it does with more samples over time).
em441 12y ago
Hi! This is Emily, the stats brains behind this. I think this is a really interesting and important question. I know this question comes up in other contexts (clinical trials) and there has been a lot of work done on how to find "stopping times" from a statistical standpoint. I'd love to see some of these statistical methods for sequential analysis (developed for clinical trials) integrated into how we do A/B testing.
tedsanders 12y ago
I totally agree. And I think Bayesian methods (which compute the likelihood of the model given the data) tend to work much better for these rolling data applications than frequentist methods (which compute the likelihood of the data given the model). The problem for frequentist methods here is that when data is constantly rolling in it's hard to specify the space of all possible data collected (because traditional bounds like the number of data points no longer work). Here's a link to a paper on a Bayesian approach to the multi-armed bandit problem: http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract;jsessionid=364E386ABCD27DA0D592DC7810CA558A.f01t03 http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract...
- em441 12y ago
  Hi! This is Emily, the stats brains behind this. I will definitely look into this and other Bayesian methods. Thanks!
  - btilly 12y ago
    I would also suggest switching from chi-square to the g-test. If you look at the history, I believe that you'll find that Pearson originally came up with the g-test as an approximation to an exact test, and then found the chi-square as an easier to compute alternative. This mattered back in the days of pencil and paper, but there is no excuse today to use the worse technique. I'm going to avoid long discussions about the advisability of taking multiple looks at results with a classical statistical test. But see my incomplete series at http://elem.com/~btilly/ab-testing-multiple-looks/index.html http://elem.com/~btilly/ab-testing-multiple-looks/index.html for some of the considerations. I never got into Bayesian statistics in there. In general they depend on the existence of a prior distribution. Careful treatments will talk about this. Sloppy ones assume one, don't talk about the one that they assume, and then quote results without letting you know about this important assumption. As long as you accept that assumption, they work well. But sometimes can be confusing to explain. (Until people "get" it. Then it can become irritating getting them to STOP explaining it!) If you want to discuss these issues more, my email is my name at gmail.com.
    - em441 12y ago
      "Upgrading" to the g-test is certainly something we could implement in the future - Pearson's Chi Square was simply a starting place. I will definitely have a look over the link you provided. Thanks!
gwern 12y ago
> There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?" Relevant keywords here include 'trial sequential analysis', 'adaptive trials', & 'multi-armed bandits'.
yummyfajitas 12y ago
This is exactly the test you are looking for: http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html It can be drastically improved in the case of emails by building a Bayesian model tuned to email itself. Most people just blindly apply a testing procedure to their situation, but if you tune the test to your situation you can make do with a LOT fewer samples.

by wtracy 12y ago

jessicaraygun 12y ago

Yes! This is very important to us in dealing with transactional email. Our Z-Test Method in Confidence.js pinpoints when results become statistically significant by calculating the required sample size at any given time using the standard error, zScore, and margin of error. If we have more samples than required, we can stop our test with confidence that we have statistically significant results. We're still working out a way to do this with the new method - we have a hunch it might have to do with identifying when the confidence level becomes stable (as it does with more samples over time).

em441 12y ago

Hi! This is Emily, the stats brains behind this. I think this is a really interesting and important question. I know this question comes up in other contexts (clinical trials) and there has been a lot of work done on how to find "stopping times" from a statistical standpoint. I'd love to see some of these statistical methods for sequential analysis (developed for clinical trials) integrated into how we do A/B testing.

tedsanders 12y ago

I totally agree. And I think Bayesian methods (which compute the likelihood of the model given the data) tend to work much better for these rolling data applications than frequentist methods (which compute the likelihood of the data given the model). The problem for frequentist methods here is that when data is constantly rolling in it's hard to specify the space of all possible data collected (because traditional bounds like the number of data points no longer work). Here's a link to a paper on a Bayesian approach to the multi-armed bandit problem: http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract;jsessionid=364E386ABCD27DA0D592DC7810CA558A.f01t03 http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract...

em441 12y ago
Hi! This is Emily, the stats brains behind this. I will definitely look into this and other Bayesian methods. Thanks!
- btilly 12y ago
  I would also suggest switching from chi-square to the g-test. If you look at the history, I believe that you'll find that Pearson originally came up with the g-test as an approximation to an exact test, and then found the chi-square as an easier to compute alternative. This mattered back in the days of pencil and paper, but there is no excuse today to use the worse technique. I'm going to avoid long discussions about the advisability of taking multiple looks at results with a classical statistical test. But see my incomplete series at http://elem.com/~btilly/ab-testing-multiple-looks/index.html http://elem.com/~btilly/ab-testing-multiple-looks/index.html for some of the considerations. I never got into Bayesian statistics in there. In general they depend on the existence of a prior distribution. Careful treatments will talk about this. Sloppy ones assume one, don't talk about the one that they assume, and then quote results without letting you know about this important assumption. As long as you accept that assumption, they work well. But sometimes can be confusing to explain. (Until people "get" it. Then it can become irritating getting them to STOP explaining it!) If you want to discuss these issues more, my email is my name at gmail.com.
  - em441 12y ago
    "Upgrading" to the g-test is certainly something we could implement in the future - Pearson's Chi Square was simply a starting place. I will definitely have a look over the link you provided. Thanks!

gwern 12y ago

> There ought to be a way to come up with adaptive thresholds such that at any given time we can ask, "Do we have statistically significant results yet, or do we keep the test running?" Relevant keywords here include 'trial sequential analysis', 'adaptive trials', & 'multi-armed bandits'.

yummyfajitas 12y ago

This is exactly the test you are looking for: http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html It can be drastically improved in the case of emails by building a Bayesian model tuned to email itself. Most people just blindly apply a testing procedure to their situation, but if you tune the test to your situation you can make do with a LOT fewer samples.