6 ms·

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll futz with things that make no difference most of the time and eventually give up. Experimentation is not about testing random permutations (unless you've got an infinite amount of time). It's about coming up with a reasonable hypothesis and then testing it. I study a web page until I come to a conclusion in my mind that something could really use some improvement. Then, even though I'm sure of it in my mind, I test it, because "sure of it in my mind" is wrong about half the time. One note to those who use Optimizely: you do need to wait at least twice as long as Optimizely thinks, because statistical significance can bounce around with too little traffic. Optimizely thought the above test was done at about 2000 views, which was far too little results to be conclusive, with only ~20 sales.

rco8786 11y ago
You really should question the statistical significance of those numbers.
- jefftk 11y ago
  Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.
  - mangeletti 11y ago
    The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
    - adevine 11y ago
      > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
      - mangeletti 11y ago
        You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
        tamana 11y ago
        Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
        xixi77 11y ago
        They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?
        norkakn 11y ago
        The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it. Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.
- deleted 11y ago
  [deleted]
- MJSplot_author 11y ago
  Running their numbers: Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better. [0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66 EDIT If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is: (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392 which shows an improvement of 1.264+-0.234. Stastically nout.
- MJSplot_author 11y ago
  The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".
- ced 11y ago
  I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.
  - CPLX 11y ago
    Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.
  - mangeletti 11y ago
    Thanks for this analysis. I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then. Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.
    - ced 11y ago
      Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.
  - wbeckler 11y ago
    I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/ http://www.peakconversion.com/2012/02/ab-split-test-graphica...

rco8786 11y ago

You really should question the statistical significance of those numbers.

jefftk 11y ago
Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.
- mangeletti 11y ago
  The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
  - adevine 11y ago
    > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
    - mangeletti 11y ago
      You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
      - tamana 11y ago
        Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
      - xixi77 11y ago
        They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?
      - norkakn 11y ago
        The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it. Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.
deleted 11y ago
[deleted]
MJSplot_author 11y ago
Running their numbers: Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better. [0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66 EDIT If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is: (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392 which shows an improvement of 1.264+-0.234. Stastically nout.
MJSplot_author 11y ago
The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".
ced 11y ago
I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.
- CPLX 11y ago
  Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.
- mangeletti 11y ago
  Thanks for this analysis. I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then. Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.
  - ced 11y ago
    Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.
- wbeckler 11y ago
  I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/ http://www.peakconversion.com/2012/02/ab-split-test-graphica...