7 ms·

Hmmh, this is interesting. Most A/B software will let you set a level of statistical confidence that needs to be attained before a winner can be declared. For e

by jkuria 10y ago

Hmmh, this is interesting. Most A/B software will let you set a level of statistical confidence that needs to be attained before a winner can be declared. For example in Google Analytics two common ones are 95% and 99%. We stop our tests when they reach at least 95% confidence. Is the author saying one must wait for 6000 events even if the difference between A/B is large? The larger the relative difference, the fewer events needed.

squarecog 10y ago
Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion. You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).
- tedsanders 10y ago
  Is it really a failure mode? I think it should be fine to stop whenever you want during a test. You just have to be smart enough to not use a t test and misinterpret it to mean than A is 95% likelier than B.
  - daveguy 10y ago
    Yes. It really is a failure mode. No, it is not fine to stop whenever you want during a test. It doesn't matter what statistical test you are using -- you need to gather enough data to have sufficient statistical power. Also, the more tests you do the lower the p-value should be. If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
    - tedsanders 10y ago
      This seems terribly dogmatic to me. Take the example of clinical trials. Suppose you're testing a new cancer drug. You design an experiment to test the new drug, named B, versus an established chemotherapy treatment named A. You expect B's performance to be similar to A's performance in controlling the cancer, so to make sure your trial has high power, you plan to test the drugs on 2,000 patients (with each drug administered to 1,000). Now consider the following two scenarios: (1) After giving drug B to 100 patients, all 100 patients are dead. Do you continue the trial, giving the (apparently) deadly drug B to 900 more patients? (2) After giving drug B to 100 patients, all 100 patients are totally cured (vs A curing 3 in 100). Do you continue the trial, withholding the (apparent) cure for cancer from 900 more patients? In either case, since you have a strong effect, it seems to me there is logical justification to end the trial early. Obviously the stakes are higher in clinical trials than website design, but in both cases, data acquisition has costs and intermediate results may inform changes to your experiment design. I honestly cannot see how anyone could blankly assert that stopping a test is always wrong. There are certainly circumstances where you do want to stop early. You just have to make sure you aren't misinterpreting a statistic when you do so.
      - apathy 10y ago
        "Wrong" isn't the word we're looking for here, I don't think. But your above example is bullshit -- nobody puts 1000 patients at risk in a phase I (safety) trial, and if the dose isn't reasonably well calibrated by the phase III study you're describing above, someone's going to jail. In Phase II we will often have stopping rules for exactly this reason, just in case the sampling was biased in the small Phase I sample. Above there are a number of things to notice: 1) The phasing approximates Thompson sampling to a degree, in that large late-phase trials MUST follow smaller early phase trials. Nobody is going to waste patients on SuperMab (look it up). 2) The endpoints are hard, fast, and pre-specified: IFF we have N adverse events in M patients, we shut down the trial for toxicity. IFF we have X or more complete responses in Y patients, we shut down the trial because it would be unethical to deprive the control arm. IFF we have Z or fewer responses in the treatment arm, given our ultimate accrual goal (total sample size), it will be impossible to conclude (using the test we have selected and preregistered) that the new drug isn't WORSE than the standard, so we'll shut it down for futility. Those patients will be better served by another trial. You are massively oversimplifying a well-understood problem. Decision theory is a thing, and it's been a thing for 100 years. Instead of lighting your strawman on fire, how about reframing it? Stopping isn't "always" wrong, but stopping because you've managed to hit some extremal value is pretty much always biased. The "Winner's curse", regression to the mean, all of these things happen because people forget about sampling variability. It's also why point estimates (even test statistics) rather than posterior distributions are misleading. If you're going to stop at an uncertain time or for unspecified reasons, you need to include the "slop" in your estimates. "We estimate that the new page is 2x (95% CI, 1.0001x-10x) more likely to result in a conversion"... hey, you stopped early and at least you're being honest about it... but if we leave out the uncertainty then it's just misleading. All of the above is taken into account when designing trials because not only do we not like killing people, we don't like going to jail for stupid avoidable mistakes.
  - imh 10y ago
    If you decide to stop when a test says the error rate (FPR) is 5%, your error rate will be higher than 5%. If you don't want to call that a failure mode, it's at least a misuse of the metric.
  - sogen 10y ago
    Yes, stopping at 95% percent is BAD. First of all you need to reach on a sample size large enough, otherwise you are just lying to yourself. Source: I have a degree in analytics.
user5994461 10y ago
> Is the author saying one must wait for 6000 events even if the difference between A/B is large? Yes. > The larger the relative difference, the fewer events needed. Wrong. That's the common mistake that is being addressed. The relative difference could be anything at any point in time. You always need a lots of events to be somewhat confident that there is really a difference. Short version: If you want X% confidence on a test, there are maths formula to say that you need to monitor N events, at least. The formula doesn't depends on how large is the difference (which you don't know anyway).
- tedsanders 10y ago
  It's absolutely true that if the difference is larger you need fewer samples to reach statistical significance. This why is why small effects require studies with large sample sizes.
- contravariant 10y ago
  Well, in this case the distribution is (assumed to be) binomial, which does allow you to assign a higher significance for cases where there is a strong effect.
contravariant 10y ago
The problem is that people often repeatedly check the significance, despite the fact that this test only guarantees a certain false positive rate if you use it once. If you're planning to stop as soon as you find a positive result you'll need to modify the tests to ensure that the total chance that any test results in a positive is low enough. In general you'll need to keep increasing the significance level as you do more events (if you only plan to test a finite number of times you can keep the significance fixed, but I think this will lead to more false negatives than an 'adaptive' significance). To illustrate, if you have 6000 events and check for 99% significance after every one, then you'd expect about 60 false positives on average. Of course these false positives aren't distributed uniformly, so it's not like you'll always find 60 false positives, however it's not like you'll only find 0 or 6000 either, meaning that (significantly) more than 1% of the time you'll have at least 1 false positive.
- tedsanders 10y ago
  Checking and stopping is only a problem if you use the inappropriate formula. It's not generally a problem as far as I'm aware. What do you think?
  - apathy 10y ago
    It is a problem. You are "spending" some of your error "budget" whenever you peek prior to an endpoint. This is why if you're going to do interim analyses in clinical trials, you have to include it in the design. People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this. If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too. People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
    - tedsanders 10y ago
daveguy 10y ago
From the article: "You can perform power calculations by using an online calculator or a statistical package like R. If time is short, a simple rule of thumb is to use 6000 conversion events in each group if you want to detect a 5% uplift. Use 1600 if you only want to detect uplifts of over 10%. These numbers will give you around 80% power to detect a true effect." These numbers assume approximately 10% of changes give true uplift (aka actual improvement). If the incidence of improvement is rarer you would need more than 6000 events to identify a 5% increase. (OT: that term sounds like cult language -- you have three chances to achieve true uplift)
paulddraper 10y ago
You can take an adaptive approach, but almost universally it is done wrong. https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... FYI, the difference with Google Analytics and most other A/B platforms is that GA uses multi-armed bandit. It adjusts the proportions according to the results. This is nice because it effectively ends the test for you, and you don't have to stress about suboptimal success rates just for the purpose of experimentation. https://support.google.com/analytics/answer/2844870 https://support.google.com/analytics/answer/2844870

by jkuria 10y ago

squarecog 10y ago

Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion. You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).

tedsanders 10y ago
Is it really a failure mode? I think it should be fine to stop whenever you want during a test. You just have to be smart enough to not use a t test and misinterpret it to mean than A is 95% likelier than B.
- daveguy 10y ago
  Yes. It really is a failure mode. No, it is not fine to stop whenever you want during a test. It doesn't matter what statistical test you are using -- you need to gather enough data to have sufficient statistical power. Also, the more tests you do the lower the p-value should be. If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
  - tedsanders 10y ago
    This seems terribly dogmatic to me. Take the example of clinical trials. Suppose you're testing a new cancer drug. You design an experiment to test the new drug, named B, versus an established chemotherapy treatment named A. You expect B's performance to be similar to A's performance in controlling the cancer, so to make sure your trial has high power, you plan to test the drugs on 2,000 patients (with each drug administered to 1,000). Now consider the following two scenarios: (1) After giving drug B to 100 patients, all 100 patients are dead. Do you continue the trial, giving the (apparently) deadly drug B to 900 more patients? (2) After giving drug B to 100 patients, all 100 patients are totally cured (vs A curing 3 in 100). Do you continue the trial, withholding the (apparent) cure for cancer from 900 more patients? In either case, since you have a strong effect, it seems to me there is logical justification to end the trial early. Obviously the stakes are higher in clinical trials than website design, but in both cases, data acquisition has costs and intermediate results may inform changes to your experiment design. I honestly cannot see how anyone could blankly assert that stopping a test is always wrong. There are certainly circumstances where you do want to stop early. You just have to make sure you aren't misinterpreting a statistic when you do so.
    - apathy 10y ago
      "Wrong" isn't the word we're looking for here, I don't think. But your above example is bullshit -- nobody puts 1000 patients at risk in a phase I (safety) trial, and if the dose isn't reasonably well calibrated by the phase III study you're describing above, someone's going to jail. In Phase II we will often have stopping rules for exactly this reason, just in case the sampling was biased in the small Phase I sample. Above there are a number of things to notice: 1) The phasing approximates Thompson sampling to a degree, in that large late-phase trials MUST follow smaller early phase trials. Nobody is going to waste patients on SuperMab (look it up). 2) The endpoints are hard, fast, and pre-specified: IFF we have N adverse events in M patients, we shut down the trial for toxicity. IFF we have X or more complete responses in Y patients, we shut down the trial because it would be unethical to deprive the control arm. IFF we have Z or fewer responses in the treatment arm, given our ultimate accrual goal (total sample size), it will be impossible to conclude (using the test we have selected and preregistered) that the new drug isn't WORSE than the standard, so we'll shut it down for futility. Those patients will be better served by another trial. You are massively oversimplifying a well-understood problem. Decision theory is a thing, and it's been a thing for 100 years. Instead of lighting your strawman on fire, how about reframing it? Stopping isn't "always" wrong, but stopping because you've managed to hit some extremal value is pretty much always biased. The "Winner's curse", regression to the mean, all of these things happen because people forget about sampling variability. It's also why point estimates (even test statistics) rather than posterior distributions are misleading. If you're going to stop at an uncertain time or for unspecified reasons, you need to include the "slop" in your estimates. "We estimate that the new page is 2x (95% CI, 1.0001x-10x) more likely to result in a conversion"... hey, you stopped early and at least you're being honest about it... but if we leave out the uncertainty then it's just misleading. All of the above is taken into account when designing trials because not only do we not like killing people, we don't like going to jail for stupid avoidable mistakes.
- imh 10y ago
  If you decide to stop when a test says the error rate (FPR) is 5%, your error rate will be higher than 5%. If you don't want to call that a failure mode, it's at least a misuse of the metric.
- sogen 10y ago
  Yes, stopping at 95% percent is BAD. First of all you need to reach on a sample size large enough, otherwise you are just lying to yourself. Source: I have a degree in analytics.

user5994461 10y ago

> Is the author saying one must wait for 6000 events even if the difference between A/B is large? Yes. > The larger the relative difference, the fewer events needed. Wrong. That's the common mistake that is being addressed. The relative difference could be anything at any point in time. You always need a lots of events to be somewhat confident that there is really a difference. Short version: If you want X% confidence on a test, there are maths formula to say that you need to monitor N events, at least. The formula doesn't depends on how large is the difference (which you don't know anyway).

tedsanders 10y ago
It's absolutely true that if the difference is larger you need fewer samples to reach statistical significance. This why is why small effects require studies with large sample sizes.
contravariant 10y ago
Well, in this case the distribution is (assumed to be) binomial, which does allow you to assign a higher significance for cases where there is a strong effect.

contravariant 10y ago

The problem is that people often repeatedly check the significance, despite the fact that this test only guarantees a certain false positive rate if you use it once. If you're planning to stop as soon as you find a positive result you'll need to modify the tests to ensure that the total chance that any test results in a positive is low enough. In general you'll need to keep increasing the significance level as you do more events (if you only plan to test a finite number of times you can keep the significance fixed, but I think this will lead to more false negatives than an 'adaptive' significance). To illustrate, if you have 6000 events and check for 99% significance after every one, then you'd expect about 60 false positives on average. Of course these false positives aren't distributed uniformly, so it's not like you'll always find 60 false positives, however it's not like you'll only find 0 or 6000 either, meaning that (significantly) more than 1% of the time you'll have at least 1 false positive.

tedsanders 10y ago
Checking and stopping is only a problem if you use the inappropriate formula. It's not generally a problem as far as I'm aware. What do you think?
- apathy 10y ago
  It is a problem. You are "spending" some of your error "budget" whenever you peek prior to an endpoint. This is why if you're going to do interim analyses in clinical trials, you have to include it in the design. People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this. If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too. People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
  - tedsanders 10y ago

daveguy 10y ago

From the article: "You can perform power calculations by using an online calculator or a statistical package like R. If time is short, a simple rule of thumb is to use 6000 conversion events in each group if you want to detect a 5% uplift. Use 1600 if you only want to detect uplifts of over 10%. These numbers will give you around 80% power to detect a true effect." These numbers assume approximately 10% of changes give true uplift (aka actual improvement). If the incidence of improvement is rarer you would need more than 6000 events to identify a 5% increase. (OT: that term sounds like cult language -- you have three chances to achieve true uplift)

paulddraper 10y ago

You can take an adaptive approach, but almost universally it is done wrong. https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... FYI, the difference with Google Analytics and most other A/B platforms is that GA uses multi-armed bandit. It adjusts the proportions according to the results. This is nice because it effectively ends the test for you, and you don't have to stress about suboptimal success rates just for the purpose of experimentation. https://support.google.com/analytics/answer/2844870 https://support.google.com/analytics/answer/2844870