4 ms·

Every time you do a significance calculation and decide whether to stop the test or continue you increase the likelihood of a type I error (i.e., false positive

by jfarmer 16y ago

Every time you do a significance calculation and decide whether to stop the test or continue you increase the likelihood of a type I error (i.e., false positive). So, your 99.8% confidence isn't 99.8%. There are a few ways to compensate for this. The easiest is to fix your sample size and not reach any conclusions until you have tested that many users. You can determine your sample size by, say, figuring out the minimum number of people needed before there's a 95% chance you observe a 10% effect size. This is a problem in clinical trials where ethical questions arise. If the test appears harmful, should we stop? If it appears beneficial, isn't it unethical to deprive the control group of the treatment? Anyhow, if you want to observe the results continuously, you need to use a technique like alpha spending, sequential experimental design, or Bayesean experimental design. TL;DR: If you're periodically looking at your A/B testing results and deciding whether to continue or stop, you're doing it wrong and your significance level is lower than you think it is.

StavrosK 16y ago
Thanks for that, that's something I haven't seen anywhere. Do you have any references that explain why this error occurs?
- carbocation 16y ago
  When you take multiple peeks at the data as it begins to accumulate, you're essentially performing multiple testing. Imagine that you wanted to see if A != B, with a 95% confidence. To oversimplify, this means that you're willing to accept that 1 in 20 times, you'll incorrectly reject the null and you will consider A different from B even though they are truly the same. If you run 20 independent tests at once, each at a 95% confidence, then by chance you'd expect 1 to reject the null even if they're actually all null. Now, if you repeatedly peek at the data as it accumulates for one test, you're doing something similar (not 100% the same because the tests aren't totally independent, but similar). To again oversimplify, you'd expect, by chance, to see "significant" results 1 in 20 of the times that you look at the data, even if there is no significant result. This is why you need to wait until you've collected all of the necessary data first, or implement a procedure to protect you from incorrectly seeing "significance" when it's not there. You might, for example, require a more stringent confidence level earlier on, and less stringent ones later.
  - StavrosK 16y ago
    Yep, that's what I realised from your and jfarmer's comments. It does make sense that you're sampling multiple times and peeking to see if you've reached that, thank you.
- ivank 16y ago
  http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://news.ycombinator.com/item?id=1277004 http://news.ycombinator.com/item?id=1277004
  - StavrosK 16y ago
    Thank you.
lionhearted 16y ago
This is one of the more insightful comments I've read lately. I thought just upvoting wasn't enough for this one, so I wanted to say thanks and I'm going to refine my next split tests along your guidelines.
Eliezer 16y ago
And this, ladies and gentlemen, is why we use Bayesian likelihood ratios. You can use whatever stopping algorithm you like on likelihood ratios and they will still be valid. "Statistical significance", not so much. A simple test program I wrote showed that with a limit of 500 coinflips, flipping a fair coin will produce, on at least one out of the steps, a dataset which rejects the null hypothesis with p < .05, fully 30% of the time. http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...
- carbocation 16y ago
  Right, but even frequentist methods can prevent an erroneous rejection of the null, if used correctly.
  - Eliezer 16y ago
    The likelihood ratio is just true no matter how you use it. The meaning of an experimental result should not depend on the experimenter's private state of mind (such as the stopping rule they followed) if that state of mind does not affect the experimental apparatus. The likelihood ratio is objective, "statistical significance" is not. See link.
    - carbocation 16y ago
      Totally agreed. But assuming that you wanted to calculate a posterior, what would you have chosen for a prior here?
      - Eliezer 16y ago
        (Note: Second paragraph in grandparent was edit, original reply in parent may have been to first paragraph only.) Well, keeping in mind that experimenters should just report likelihood ratios, I'd have estimated a pretty low prior probability that one of two identical pages produced more signups than the other. But I suppose that after seeing a likelihood ratio of 1000/1 favoring the hypothesis "each of these pages has a separate fixed conversion rate" over "both of these pages have the same fixed conversion rate", using Laplace's Rule of Succession (i.e. a flat prior between 0 and 1 for the conversion rate) in all cases, I'd think that something was more likely than not going on.
SemanticFog 16y ago
Just to clarify this entirely correct analysis: It's not a "peeking" error, it's a "stopping" error. As long as you decide on a minimum number of trials to make a decision, it's fine to look at the data as it is accumulating.
SemanticFog 16y ago
It's also worth pointing out that you can safely peek and make an early decision if you insist on a higher confidence interval for early stops. The exact numbers depend on how often you test the confidence interval, but a multiple of 5-10 is for early measures will usually give reasonable results. E.g., let's say you ultimately want a 98% confidence interval (2% chance of error). A 10x improvement is a .2% change of error, or 99.8%. Depending on how often you are sampling, an early result with 99.8% confidence can be equally accurate as a later results with 98% confidence.

by jfarmer 16y ago

StavrosK 16y ago

Thanks for that, that's something I haven't seen anywhere. Do you have any references that explain why this error occurs?

carbocation 16y ago
When you take multiple peeks at the data as it begins to accumulate, you're essentially performing multiple testing. Imagine that you wanted to see if A != B, with a 95% confidence. To oversimplify, this means that you're willing to accept that 1 in 20 times, you'll incorrectly reject the null and you will consider A different from B even though they are truly the same. If you run 20 independent tests at once, each at a 95% confidence, then by chance you'd expect 1 to reject the null even if they're actually all null. Now, if you repeatedly peek at the data as it accumulates for one test, you're doing something similar (not 100% the same because the tests aren't totally independent, but similar). To again oversimplify, you'd expect, by chance, to see "significant" results 1 in 20 of the times that you look at the data, even if there is no significant result. This is why you need to wait until you've collected all of the necessary data first, or implement a procedure to protect you from incorrectly seeing "significance" when it's not there. You might, for example, require a more stringent confidence level earlier on, and less stringent ones later.
- StavrosK 16y ago
  Yep, that's what I realised from your and jfarmer's comments. It does make sense that you're sampling multiple times and peeking to see if you've reached that, thank you.
ivank 16y ago
http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://news.ycombinator.com/item?id=1277004 http://news.ycombinator.com/item?id=1277004
- StavrosK 16y ago
  Thank you.

lionhearted 16y ago

This is one of the more insightful comments I've read lately. I thought just upvoting wasn't enough for this one, so I wanted to say thanks and I'm going to refine my next split tests along your guidelines.

Eliezer 16y ago

And this, ladies and gentlemen, is why we use Bayesian likelihood ratios. You can use whatever stopping algorithm you like on likelihood ratios and they will still be valid. "Statistical significance", not so much. A simple test program I wrote showed that with a limit of 500 coinflips, flipping a fair coin will produce, on at least one out of the steps, a dataset which rejects the null hypothesis with p < .05, fully 30% of the time. http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

carbocation 16y ago
Right, but even frequentist methods can prevent an erroneous rejection of the null, if used correctly.
- Eliezer 16y ago
  The likelihood ratio is just true no matter how you use it. The meaning of an experimental result should not depend on the experimenter's private state of mind (such as the stopping rule they followed) if that state of mind does not affect the experimental apparatus. The likelihood ratio is objective, "statistical significance" is not. See link.
  - carbocation 16y ago
    Totally agreed. But assuming that you wanted to calculate a posterior, what would you have chosen for a prior here?
    - Eliezer 16y ago
      (Note: Second paragraph in grandparent was edit, original reply in parent may have been to first paragraph only.) Well, keeping in mind that experimenters should just report likelihood ratios, I'd have estimated a pretty low prior probability that one of two identical pages produced more signups than the other. But I suppose that after seeing a likelihood ratio of 1000/1 favoring the hypothesis "each of these pages has a separate fixed conversion rate" over "both of these pages have the same fixed conversion rate", using Laplace's Rule of Succession (i.e. a flat prior between 0 and 1 for the conversion rate) in all cases, I'd think that something was more likely than not going on.

SemanticFog 16y ago

Just to clarify this entirely correct analysis: It's not a "peeking" error, it's a "stopping" error. As long as you decide on a minimum number of trials to make a decision, it's fine to look at the data as it is accumulating.

SemanticFog 16y ago

It's also worth pointing out that you can safely peek and make an early decision if you insist on a higher confidence interval for early stops. The exact numbers depend on how often you test the confidence interval, but a multiple of 5-10 is for early measures will usually give reasonable results. E.g., let's say you ultimately want a 98% confidence interval (2% chance of error). A 10x improvement is a .2% change of error, or 99.8%. Depending on how often you are sampling, an early result with 99.8% confidence can be equally accurate as a later results with 98% confidence.