4 ms·

This post would be useful if it told us (1) the actual numbers of visitors vs successes along the timepoints that you mention, and (2) the formula that you used

This post would be useful if it told us (1) the actual numbers of visitors vs successes along the timepoints that you mention, and (2) the formula that you used to calculate confidence. Without knowing those 2 facts, it's hard to conclude anything. Also, is that confidence interval corrected for the multiple peeks that you took at the data? Etc. I'd like to know much more about the particulars of the methodology here, because it sounds like statistical methods might have been misunderstood or misused (perhaps by the A/B software or anywhere else along the chain).

StavrosK 16y ago
I can't share the actual data, but the method of calculating the confidence was the standard formula from calculating the confidence for the Z-score as used in http://abtester.com/calculator/ http://abtester.com/calculator/, Google Website optimiser, etc etc. The numbers did check out, it was just an unlikely fluke, which is why I don't trust the confidence interval as much as I used to...
- carbocation 16y ago
  Before running this test, did you estimate the sample size needed to detect the estimated difference in effects from A vs B? Since the expected difference is 0, the necessary sample size would be infinite. Therefore, we could at least estimate a very small effect, which would necessitate a very large sample. I would be very cautious, then, about trying to interpret the "confidence interval" before you had accumulated that large sample size. If you're going to peek before getting to that point, then you should almost certainly have implemented an alpha spending function (or similar) in order to maintain the desired overall error rate. This should not reduce your belief in confidence intervals; this is a great, motivating opportunity that should prompt you to learn how to use them correctly.
  - StavrosK 16y ago
    No, we hadn't estimated the necessary sample size. To be honest, nobody expected the confidence metric to go so high for a test with no changes, which is why we were surprised. We've studied various A/B testing resources and we haven't seen this "peeking error" mentioned anywhere, so we'd appreciate any resources you may have (and would possibly summarise them in a subsequent post). Thanks for the help!
    - carbocation 16y ago
      Sure thing. I just posted a bit more detail about this elsewhere in this thread. Also, googling "alpha spending function" brings back some useful results. (I can't paste into the HN text box with the latest Chrome dev, otherwise I'd paste some links here for you.)
      - StavrosK 16y ago
        I saw that as well, thanks. This isn't the sort of statistics we did in uni, so it's always good to get extra pointers. Thanks again!

StavrosK 16y ago

I can't share the actual data, but the method of calculating the confidence was the standard formula from calculating the confidence for the Z-score as used in http://abtester.com/calculator/ http://abtester.com/calculator/, Google Website optimiser, etc etc. The numbers did check out, it was just an unlikely fluke, which is why I don't trust the confidence interval as much as I used to...

carbocation 16y ago
Before running this test, did you estimate the sample size needed to detect the estimated difference in effects from A vs B? Since the expected difference is 0, the necessary sample size would be infinite. Therefore, we could at least estimate a very small effect, which would necessitate a very large sample. I would be very cautious, then, about trying to interpret the "confidence interval" before you had accumulated that large sample size. If you're going to peek before getting to that point, then you should almost certainly have implemented an alpha spending function (or similar) in order to maintain the desired overall error rate. This should not reduce your belief in confidence intervals; this is a great, motivating opportunity that should prompt you to learn how to use them correctly.
- StavrosK 16y ago
  No, we hadn't estimated the necessary sample size. To be honest, nobody expected the confidence metric to go so high for a test with no changes, which is why we were surprised. We've studied various A/B testing resources and we haven't seen this "peeking error" mentioned anywhere, so we'd appreciate any resources you may have (and would possibly summarise them in a subsequent post). Thanks for the help!
  - carbocation 16y ago
    Sure thing. I just posted a bit more detail about this elsewhere in this thread. Also, googling "alpha spending function" brings back some useful results. (I can't paste into the HN text box with the latest Chrome dev, otherwise I'd paste some links here for you.)
    - StavrosK 16y ago
      I saw that as well, thanks. This isn't the sort of statistics we did in uni, so it's always good to get extra pointers. Thanks again!