5 ms·

You really should question the statistical significance of those numbers.

by rco8786 11y ago

jefftk 11y ago
Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.
- mangeletti 11y ago
  The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
  - adevine 11y ago
    > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
    - mangeletti 11y ago
      You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
      - tamana 11y ago
        Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
      - xixi77 11y ago
        They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?
      - norkakn 11y ago
        The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it. Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.
        gingerlime 11y ago
        > but they never really published much justifying it Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...). I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course. [0] http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf http://pages.optimizely.com/rs/optimizely/images/stats_engin...
        norkakn 11y ago
        Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results. They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people. What are your reasons for disliking Optimizely?
      - Quinner 11y ago
        There's a reason you'll only see their number say >99%
      - gk1 11y ago
        You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page. Being smug and condescending really backfires when you don't know what you're talking about.
        mangeletti 11y ago
        Bro... http://cl.ly/1b0B3Y3o1w09 http://cl.ly/1b0B3Y3o1w09 > Being smug and condescending really backfires when you don't know what you're talking about. How's that working out for you?
deleted 11y ago
[deleted]
MJSplot_author 11y ago
Running their numbers: Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better. [0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66 EDIT If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is: (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392 which shows an improvement of 1.264+-0.234. Stastically nout.
MJSplot_author 11y ago
The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".
ced 11y ago
I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.
- CPLX 11y ago
  Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.
- mangeletti 11y ago
  Thanks for this analysis. I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then. Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.
  - ced 11y ago
    Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.
- wbeckler 11y ago
  I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/ http://www.peakconversion.com/2012/02/ab-split-test-graphica...

You really should question the statistical significance of those numbers.

by rco8786 11y ago

You really should question the statistical significance of those numbers.

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.

mangeletti 11y ago
The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
- adevine 11y ago
  > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
  - mangeletti 11y ago
    You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
    - tamana 11y ago
      Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
    - xixi77 11y ago
      They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?
    - norkakn 11y ago
      The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it. Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.
      - gingerlime 11y ago
        > but they never really published much justifying it Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...). I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course. [0] http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf http://pages.optimizely.com/rs/optimizely/images/stats_engin...
        norkakn 11y ago
        Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results. They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people. What are your reasons for disliking Optimizely?
    - Quinner 11y ago
      There's a reason you'll only see their number say >99%
    - gk1 11y ago
      You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page. Being smug and condescending really backfires when you don't know what you're talking about.
      - mangeletti 11y ago
        Bro... http://cl.ly/1b0B3Y3o1w09 http://cl.ly/1b0B3Y3o1w09 > Being smug and condescending really backfires when you don't know what you're talking about. How's that working out for you?

deleted 11y ago

[deleted]

MJSplot_author 11y ago

Running their numbers: Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better. [0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66 EDIT If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is: (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392 which shows an improvement of 1.264+-0.234. Stastically nout.

MJSplot_author 11y ago

The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".

ced 11y ago

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.

CPLX 11y ago
Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.
mangeletti 11y ago
Thanks for this analysis. I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then. Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.
- ced 11y ago
  Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.
wbeckler 11y ago
I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/ http://www.peakconversion.com/2012/02/ab-split-test-graphica...