4 ms·

Appsumo reveals its A/B testing secret: only 1 out of 8 tests produce results

44 points by paraschopra 16y ago

patio11 16y ago
This is roughly my experience, too -- take a look at how many null results ("no significant difference") I rack up: http://www.bingocardcreator.com/abingo/results http://www.bingocardcreator.com/abingo/results To add insult to injury, half of the remainder were significant... in the wrong direction. (Edit to add: null results aren't failures, though. What is the Edison quote: you now know one more thing that didn't work.)
- paraschopra 16y ago
  Yup, same experience with me too (while observing customers' tests and doing my own A/B tests). Noah provided great counter-examples. I mostly publish only positive A/B test results as case studies but this doesn't mean every A/B test produces spectacular results.
- tomjen3 16y ago
  I wouldn't call them failed tests. In science a null result mean that you didn't learn anything but in this case who cares if p is 0.6? It is still more likely to make you money.
  - barrkel 16y ago
    I think precisely the point of not being able to reject the null hypothesis is that you can't be sure it's more likely to make you more money; otherwise you could have required a lower confidence level and still rejected the null hypothesis.
  - bdonlan 16y ago
    The thing about non-statistically-significant tests is you can't, in fact, say "it is still more likely to make you money." You can say that the B set did in fact make you more money this time, but you can't rule out that being luck of the draw. It might even have a (very small) bias AGAINST making you money, but that bias got lost in the noise. That's why it's important to discard non-statistically-significant tests (or expand the sample size if you _really_ want to know...)
- tel 16y ago
  Testing involves paying a price in example counts in order to buy analytical power at some unknown rate. Insignificant results arise when either (1) people react with a lot of variance and that power/example rate is small or (2) the effect you're trying to analyze is so tiny that you need a great deal of power to see it at all. Unfortunately, it's difficult to determine the difference between those two cases. If many of your tests are failing to be significant, it could be that you're simply never investing enough to get the power you need: your users are somewhat inflexible to the design changes you're making. It could also be that many of your changes simply are pretty insignificant and by the time you wait a full year gathering simply infinite numbers of impressions, you'll find it was just a waste of time. Again, practice statistics with a great deal of self-awareness. They're only meant to inform.
wccrawford 16y ago
Every test produces results. Only 1 in 8 produce positive results. Knowing that 2 different approaches are approximately the same is a great help in design... It eliminates that 'Which way is better?' worry and lets you design it the way that looks best, instead.
- paraschopra 16y ago
  Yep, it is true that even insignificant results can help you concentrating on how to make it look better. But insignificant results are disappointment if your goal is to increase conversion rate.
ig1 16y ago
These aren't failures, if the A/B test shows no impact or that the change is a negative, then both of those add value in telling you what to concentrate on. Before people did A/B test, it was generally assumed on Facebook ads that the title was the second most important factor on CTR so people spent a lot of time tuning title copy. After lots of people did A/B testing it turns out that title copy has almost no impact, people switched their title to Chinese (for a non-chinese speaking audience) without seeing any change. That "no significant difference" means that thousands of man hours can be spent elsewhere rather than copywriting ad titles.
- paraschopra 16y ago
  I'd be wary of concluding too much from other people's A/B tests. Yes, they serve as a good starting point but doing A/B tests yourself is still imperative. Case in point: Noah found out headlines didn't matter to his audience. But for many of our customers, a small change in headline does produce significant results.
  - ig1 16y ago
    Sure, I wasn't saying that the headline results were generalizable, but "no significant difference" A/B testing results can tell you to stop wasting time in an area.
noelwelsh 16y ago
A lot of these "no results" are just bad statistics. When a test shows no result it does not mean there is no significant difference. It means that the test cannot reliably tell given the available data. Technically, this is because as you decrease the probability of a type I error (lower the p-value) you increase the probability of a type II error. To lower the probability of a type II error, and hence have null results actually mean there is no significant difference, you needs lots of hits. Under assumptions that are reasonable for most e-commerce sites (e.g. 5% conversion, you want to distinguish changes of 5%) you need about 100'000 hits. Very few people have this much data, and hence their inferences are flawed. You can do a lot better than A/B testing with better maths. More here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...
- paraschopra 16y ago
  I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X power and Y statistical significance cutoff. We have a calculator here to help determine what is the maximum number of visitors you need before giving up on a test: http://visualwebsiteoptimizer.com/ab-split-test-duration/ http://visualwebsiteoptimizer.com/ab-split-test-duration/
  - noelwelsh 16y ago
    That's a pretty big assumption to make. I've seen people doing A/B testing w/ a few thousand hits max, which is not enough for most scenarios.
    - paraschopra 16y ago
      What's the big assumption? We are using math here, not heuristics. There is no thumb rule like thousands and hundreds of visitors needed to get significant results.
      - noelwelsh 16y ago
eljaco 16y ago
They might also want to test a link to an FAQ or Details page. I was trying to see if this was only for iPhone or Android but there's really only one option : give us your email.
- noahkagan 16y ago
  Good point. It's mostly for web or client (Mac / Windows). Might do deals on phone apps in the future. http://appsumo.com/faq http://appsumo.com/faq
gohat 16y ago
I'm not sure these examples are entirely failed tests - rather they teach what is working and what isn't.
- noahkagan 16y ago
  That's really true. It's funny that some of the changes we weren't expecting to make much of an impact have had the biggest results.
random42 16y ago
Negative results are results too.

44 points by paraschopra 16y ago

This is roughly my experience, too -- take a look at how many null results ("no significant difference") I rack up: http://www.bingocardcreator.com/abingo/results http://www.bingocardcreator.com/abingo/results To add insult to injury, half of the remainder were significant... in the wrong direction. (Edit to add: null results aren't failures, though. What is the Edison quote: you now know one more thing that didn't work.)

paraschopra 16y ago
Yup, same experience with me too (while observing customers' tests and doing my own A/B tests). Noah provided great counter-examples. I mostly publish only positive A/B test results as case studies but this doesn't mean every A/B test produces spectacular results.
tomjen3 16y ago
I wouldn't call them failed tests. In science a null result mean that you didn't learn anything but in this case who cares if p is 0.6? It is still more likely to make you money.
- barrkel 16y ago
  I think precisely the point of not being able to reject the null hypothesis is that you can't be sure it's more likely to make you more money; otherwise you could have required a lower confidence level and still rejected the null hypothesis.
- bdonlan 16y ago
  The thing about non-statistically-significant tests is you can't, in fact, say "it is still more likely to make you money." You can say that the B set did in fact make you more money this time, but you can't rule out that being luck of the draw. It might even have a (very small) bias AGAINST making you money, but that bias got lost in the noise. That's why it's important to discard non-statistically-significant tests (or expand the sample size if you _really_ want to know...)
tel 16y ago
Testing involves paying a price in example counts in order to buy analytical power at some unknown rate. Insignificant results arise when either (1) people react with a lot of variance and that power/example rate is small or (2) the effect you're trying to analyze is so tiny that you need a great deal of power to see it at all. Unfortunately, it's difficult to determine the difference between those two cases. If many of your tests are failing to be significant, it could be that you're simply never investing enough to get the power you need: your users are somewhat inflexible to the design changes you're making. It could also be that many of your changes simply are pretty insignificant and by the time you wait a full year gathering simply infinite numbers of impressions, you'll find it was just a waste of time. Again, practice statistics with a great deal of self-awareness. They're only meant to inform.

wccrawford 16y ago

Every test produces results. Only 1 in 8 produce positive results. Knowing that 2 different approaches are approximately the same is a great help in design... It eliminates that 'Which way is better?' worry and lets you design it the way that looks best, instead.

paraschopra 16y ago
Yep, it is true that even insignificant results can help you concentrating on how to make it look better. But insignificant results are disappointment if your goal is to increase conversion rate.

ig1 16y ago

These aren't failures, if the A/B test shows no impact or that the change is a negative, then both of those add value in telling you what to concentrate on. Before people did A/B test, it was generally assumed on Facebook ads that the title was the second most important factor on CTR so people spent a lot of time tuning title copy. After lots of people did A/B testing it turns out that title copy has almost no impact, people switched their title to Chinese (for a non-chinese speaking audience) without seeing any change. That "no significant difference" means that thousands of man hours can be spent elsewhere rather than copywriting ad titles.

paraschopra 16y ago
I'd be wary of concluding too much from other people's A/B tests. Yes, they serve as a good starting point but doing A/B tests yourself is still imperative. Case in point: Noah found out headlines didn't matter to his audience. But for many of our customers, a small change in headline does produce significant results.
- ig1 16y ago
  Sure, I wasn't saying that the headline results were generalizable, but "no significant difference" A/B testing results can tell you to stop wasting time in an area.

noelwelsh 16y ago

A lot of these "no results" are just bad statistics. When a test shows no result it does not mean there is no significant difference. It means that the test cannot reliably tell given the available data. Technically, this is because as you decrease the probability of a type I error (lower the p-value) you increase the probability of a type II error. To lower the probability of a type II error, and hence have null results actually mean there is no significant difference, you needs lots of hits. Under assumptions that are reasonable for most e-commerce sites (e.g. 5% conversion, you want to distinguish changes of 5%) you need about 100'000 hits. Very few people have this much data, and hence their inferences are flawed. You can do a lot better than A/B testing with better maths. More here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...

paraschopra 16y ago
I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X power and Y statistical significance cutoff. We have a calculator here to help determine what is the maximum number of visitors you need before giving up on a test: http://visualwebsiteoptimizer.com/ab-split-test-duration/ http://visualwebsiteoptimizer.com/ab-split-test-duration/
- noelwelsh 16y ago
  That's a pretty big assumption to make. I've seen people doing A/B testing w/ a few thousand hits max, which is not enough for most scenarios.
  - paraschopra 16y ago
    What's the big assumption? We are using math here, not heuristics. There is no thumb rule like thousands and hundreds of visitors needed to get significant results.
    - noelwelsh 16y ago

eljaco 16y ago

They might also want to test a link to an FAQ or Details page. I was trying to see if this was only for iPhone or Android but there's really only one option : give us your email.

noahkagan 16y ago
Good point. It's mostly for web or client (Mac / Windows). Might do deals on phone apps in the future. http://appsumo.com/faq http://appsumo.com/faq

gohat 16y ago

I'm not sure these examples are entirely failed tests - rather they teach what is working and what isn't.

noahkagan 16y ago
That's really true. It's funny that some of the changes we weren't expecting to make much of an impact have had the biggest results.

random42 16y ago

Negative results are results too.