7 ms·

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color /

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / apps really have >100K uniques? If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc. And if you have <100K uniques, does the increase in conversion pay for the amount of time you have invested in setting up the test? In my experience, only when you test some completely different pages you'll see some significant differences.

mangeletti 11y ago
> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll futz with things that make no difference most of the time and eventually give up. Experimentation is not about testing random permutations (unless you've got an infinite amount of time). It's about coming up with a reasonable hypothesis and then testing it. I study a web page until I come to a conclusion in my mind that something could really use some improvement. Then, even though I'm sure of it in my mind, I test it, because "sure of it in my mind" is wrong about half the time. One note to those who use Optimizely: you do need to wait at least twice as long as Optimizely thinks, because statistical significance can bounce around with too little traffic. Optimizely thought the above test was done at about 2000 views, which was far too little results to be conclusive, with only ~20 sales.
- rco8786 11y ago
  You really should question the statistical significance of those numbers.
  - jefftk 11y ago
    Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.
    - mangeletti 11y ago
      The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
      - adevine 11y ago
        > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
        mangeletti 11y ago
        You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
        tamana 11y ago
        Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
        xixi77 11y ago
teekert 11y ago
" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc." I'd serve them at the same time, randomize which clients see which one.
- chii 11y ago
  And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...
  - eru 11y ago
    That's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.
- spiderfarmer 11y ago
  Ofcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098 https://news.ycombinator.com/item?id=11438098
butler14 11y ago
seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.
- spiderfarmer 11y ago
  Let's assume your site is now working with option A. Conversion rate is 5.7%, measured over a month. You're now running an A/B for 1 week with option A and another version, option B. You get a conversion rate of 6.5% for option A and 5.9% for option B. Normally, you'd say that 6.5% is better than 5.9%. But how sure can you be if you don't control the other factors and both are performing better than before? How many visitors do you need to offset the influence of offsite factors?
  - cbr 11y ago
    Set your site up to be able to handle option A and option B at the same time, randomly assigning visitors to each. Now you don't need to control for anything, because you can just compare the A-visitors to the B-visitors. In your case, option A started doing a lot better when you started running the test, which is moderately surprising, so maybe you messed something up in user assignment, tracking, or something. But if your framework is good, then it sounds like an external change. You still want to look at how A's rate and B's rate compare for the time period when you were randomizing, so 6.5% vs 5.9%, with the earlier 5.7% being more or less irrelevant. You do need to do significance testing to see how likely you would be to get this wide a difference by chance. The easiest way to do that is to enter your #samples and #conversions into a significance calculator [1], and the p-value tells you how likely this is (assuming you didn't have any prior reason to expect one side of the experiment to be any better). [1] Like this one https://vwo.com/ab-split-test-significance-calculator/ https://vwo.com/ab-split-test-significance-calculator/ , but multiply the p-values it gives you by 2 to account for them incorrectly using a one-tailed test.
- KingMob 11y ago
  To ignore seasonality requires assuming it has roughly the same effect on both groups. If they move in different directions or by very different amounts, you can't actually ignore it. Simple example: you're comparing your standard site's theme with a new pumpkin orange theme. In October, the pumpkin orange theme might go great, whereas in December, it might perform worse. There's clearly a seasonality interaction you need to account for. What you're thinking of is more like testing e-commerce button colors right after Christmas. After Christmas, it's likely that all your groups perform worse, but equally so.
darkxanthos 11y ago
My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business." You have better things to do.
- spiderfarmer 11y ago
  I agree, every article about A/B testing is talking about making small changes and measure them, while you should be measuring only big changes or nothing at all.
  - deleted 11y ago
    [deleted]
AJ007 11y ago
You are right, and the guy that responded with a test of just under 150 conversions is a great example of exactly what people get wrong. I have been doing design optimization for nearly a decade now and virtually every example I've seen made public is incredibly poor. Sometimes they do not even include any numbers, but just say, look first I had a 1% conversion rate and now I got 1.2%! I wish I had a good example of optimization case study, but I don't recall the last time I saw one. Without giving anything away, I have done tests this year which required over a million impressions to find out in a statistically significant way not to make any changes. My general theory is that a never optimized design with a confusing UI has a lot of low hanging fruit. You start cleaning up the bad elements and conversion rates can double or triple. Even if the sample size is less than ideal, these really big pops will be apparent. Design knowledge is better now than it was in 2003. Mobile forces designers to use one column and leave out a lot of crap. There are a lot of good examples that get copied and good out of the box software. That means when you start optimizing, the low hanging fruit is gone and you need really big sample sizes. Once the low hanging fruit is gone, often those big samples just tell you not to make any changes. Thinking about free-to-play mobile games recently (an area I have no experience in.) If 1% or fewer of users are converting you really do need a huge install base to beat your competitors at optimization. You need millions of users just to get to 20,000 or 30,000 paying players to test behavioral changes on. That means there actually is some staying power for the winners, at least for a while.
occamrazor 11y ago
People read about Google testing different shades of blue on their home page and want to do the same on their website. They tend to forget that Google has a billion unique users per day, and their website a few hundreds.

mangeletti 11y ago

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll futz with things that make no difference most of the time and eventually give up. Experimentation is not about testing random permutations (unless you've got an infinite amount of time). It's about coming up with a reasonable hypothesis and then testing it. I study a web page until I come to a conclusion in my mind that something could really use some improvement. Then, even though I'm sure of it in my mind, I test it, because "sure of it in my mind" is wrong about half the time. One note to those who use Optimizely: you do need to wait at least twice as long as Optimizely thinks, because statistical significance can bounce around with too little traffic. Optimizely thought the above test was done at about 2000 views, which was far too little results to be conclusive, with only ~20 sales.

rco8786 11y ago
You really should question the statistical significance of those numbers.
- jefftk 11y ago
  Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.
  - mangeletti 11y ago
    The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.
    - adevine 11y ago
      > I wanted Optimizely to say it was 100% significant That's not how statistical significance works...
      - mangeletti 11y ago
        You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
        tamana 11y ago
        Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy. Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.
        xixi77 11y ago

teekert 11y ago

" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc." I'd serve them at the same time, randomize which clients see which one.

chii 11y ago
And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...
- eru 11y ago
  That's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.
spiderfarmer 11y ago
Ofcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098 https://news.ycombinator.com/item?id=11438098

butler14 11y ago

seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.

spiderfarmer 11y ago
Let's assume your site is now working with option A. Conversion rate is 5.7%, measured over a month. You're now running an A/B for 1 week with option A and another version, option B. You get a conversion rate of 6.5% for option A and 5.9% for option B. Normally, you'd say that 6.5% is better than 5.9%. But how sure can you be if you don't control the other factors and both are performing better than before? How many visitors do you need to offset the influence of offsite factors?
- cbr 11y ago
  Set your site up to be able to handle option A and option B at the same time, randomly assigning visitors to each. Now you don't need to control for anything, because you can just compare the A-visitors to the B-visitors. In your case, option A started doing a lot better when you started running the test, which is moderately surprising, so maybe you messed something up in user assignment, tracking, or something. But if your framework is good, then it sounds like an external change. You still want to look at how A's rate and B's rate compare for the time period when you were randomizing, so 6.5% vs 5.9%, with the earlier 5.7% being more or less irrelevant. You do need to do significance testing to see how likely you would be to get this wide a difference by chance. The easiest way to do that is to enter your #samples and #conversions into a significance calculator [1], and the p-value tells you how likely this is (assuming you didn't have any prior reason to expect one side of the experiment to be any better). [1] Like this one https://vwo.com/ab-split-test-significance-calculator/ https://vwo.com/ab-split-test-significance-calculator/ , but multiply the p-values it gives you by 2 to account for them incorrectly using a one-tailed test.
KingMob 11y ago
To ignore seasonality requires assuming it has roughly the same effect on both groups. If they move in different directions or by very different amounts, you can't actually ignore it. Simple example: you're comparing your standard site's theme with a new pumpkin orange theme. In October, the pumpkin orange theme might go great, whereas in December, it might perform worse. There's clearly a seasonality interaction you need to account for. What you're thinking of is more like testing e-commerce button colors right after Christmas. After Christmas, it's likely that all your groups perform worse, but equally so.

darkxanthos 11y ago

My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business." You have better things to do.

spiderfarmer 11y ago
I agree, every article about A/B testing is talking about making small changes and measure them, while you should be measuring only big changes or nothing at all.
- deleted 11y ago
  [deleted]

AJ007 11y ago

You are right, and the guy that responded with a test of just under 150 conversions is a great example of exactly what people get wrong. I have been doing design optimization for nearly a decade now and virtually every example I've seen made public is incredibly poor. Sometimes they do not even include any numbers, but just say, look first I had a 1% conversion rate and now I got 1.2%! I wish I had a good example of optimization case study, but I don't recall the last time I saw one. Without giving anything away, I have done tests this year which required over a million impressions to find out in a statistically significant way not to make any changes. My general theory is that a never optimized design with a confusing UI has a lot of low hanging fruit. You start cleaning up the bad elements and conversion rates can double or triple. Even if the sample size is less than ideal, these really big pops will be apparent. Design knowledge is better now than it was in 2003. Mobile forces designers to use one column and leave out a lot of crap. There are a lot of good examples that get copied and good out of the box software. That means when you start optimizing, the low hanging fruit is gone and you need really big sample sizes. Once the low hanging fruit is gone, often those big samples just tell you not to make any changes. Thinking about free-to-play mobile games recently (an area I have no experience in.) If 1% or fewer of users are converting you really do need a huge install base to beat your competitors at optimization. You need millions of users just to get to 20,000 or 30,000 paying players to test behavioral changes on. That means there actually is some staying power for the winners, at least for a while.

occamrazor 11y ago

People read about Google testing different shades of blue on their home page and want to do the same on their website. They tend to forget that Google has a billion unique users per day, and their website a few hundreds.