5 ms·

I own a subscription-based online business, and I've done many A/B tests of new designs, and my conclusions is pretty similar to yours. Namely, it's always way

by jseims 14y ago

I own a subscription-based online business, and I've done many A/B tests of new designs, and my conclusions is pretty similar to yours. Namely, it's always way more work than you expect, especially when you have legacy customers with prior expectations. And results hardly ever budge. My hypothesis is A/B testing can move the needle for light engagement, like "try a free trial". But pulling out a credit card requires a lot of motivation, and the 0.1% of your visitors that have this motivation are relatively unaffected by your design.

patio11 14y ago
Any particular change (like this redesign) might not move the needle, but there exists SUBSTANTIAL evidence that A/B testing can help the bottom line of the company. Read e.g. any of the multiple times the 37 Signals blog covered how they got like 40% lifts to paid account signups by redoing a pricing or landing page. Do you find me credible on this topic? I'm legally, morally, and practically constrained regarding how much detail I can go into here, but if you trust me: I have recent experience which makes me disagree in the strongest possible way with the generalization of your hypothesis. A recent client CCs me on their weekly Optimizely status report. I've been evangelizing A/B testing for a few years. Every time I read one of those emails my mind gets blown again.
- tomjen3 14y ago
  But wouldn't there be big gains initially and then, as the easy fruit was plucked, the changes would give less and less and the successful tests would become fewer and fewer?
  - lincolnq 14y ago
    Likely yes. But what matters is return on time in contrast to any of the other things you could be doing. "always work on the most important thing". Even small percentage point improvements in conversion go directly to the bottom line, which is far from obviously the case when fixing bugs or adding features. And if your bottom line is already pretty big, the marginal value of someone working on optimization is likely to continue to be high, regardless of how plucked the fruit seems to be.
- jseims 14y ago
  I'm sure others have achieved significant gains... I just haven't experienced them, and it sounds like you haven't, either. What I found does move the needle are: 1) Changing how much I charge, including adding upsells, and 2) Getting new sources of traffic.
  - noahc 14y ago
    Sounds like you havent either.... That's like telling the director of the FBI he knows nothing about law enforcement because he wont reveal facts that he has a professional obligation to not reveal.
- femto 14y ago
  Are you aware of anyone experimenting with the automatic evolution of web pages? For example, use genetic programming to periodically modify the page served up. Both pages are served up, A/B style, and the page which has the highest metric (eg. $ earned) survives. Repeat. It would be interesting to see what the results are after an extended period of time.
  - Estragon 14y ago
    Not quite what you're asking about, but an interesting related innovation: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...
  - zackzackzack 14y ago
    Having played with genetic algorithms before, that is a hugely interesting idea to me. You would need to take special care on the types of mutations used though. I don't think it is possible to do it well at scale, but it would be very impressive nonetheless.
adrianhoward 14y ago
My hypothesis is A/B testing can move the needle for light engagement, like "try a free trial". But pulling out a credit card requires a lot of motivation, and the 0.1% of your visitors that have this motivation are relatively unaffected by your design. That's not true in my experience. I've seen some big increases in the percentages that complete the checkout process through an iterative process of redesign/test for example. The motivation behind a user purchase can be very different in different situations. If Mary is purchasing a new gearbox for the car, and she's already done a comparison shop, then she's likely to finish the purchase as long as it's vaguely sane. If Bob is impulse buying that new laptop bag he saw on swissmiss then every single grain of friction between Bob and the final "purchase" button is going to make it more likely that he remembers he really needs that $80 for groceries this week. At some point you're going to reach a maximum of course and only see small or no increases - but, with my clients anyway, folk seem to be quite a long way from that in many instances :-)

by jseims 14y ago

patio11 14y ago

Any particular change (like this redesign) might not move the needle, but there exists SUBSTANTIAL evidence that A/B testing can help the bottom line of the company. Read e.g. any of the multiple times the 37 Signals blog covered how they got like 40% lifts to paid account signups by redoing a pricing or landing page. Do you find me credible on this topic? I'm legally, morally, and practically constrained regarding how much detail I can go into here, but if you trust me: I have recent experience which makes me disagree in the strongest possible way with the generalization of your hypothesis. A recent client CCs me on their weekly Optimizely status report. I've been evangelizing A/B testing for a few years. Every time I read one of those emails my mind gets blown again.

tomjen3 14y ago
But wouldn't there be big gains initially and then, as the easy fruit was plucked, the changes would give less and less and the successful tests would become fewer and fewer?
- lincolnq 14y ago
  Likely yes. But what matters is return on time in contrast to any of the other things you could be doing. "always work on the most important thing". Even small percentage point improvements in conversion go directly to the bottom line, which is far from obviously the case when fixing bugs or adding features. And if your bottom line is already pretty big, the marginal value of someone working on optimization is likely to continue to be high, regardless of how plucked the fruit seems to be.
jseims 14y ago
I'm sure others have achieved significant gains... I just haven't experienced them, and it sounds like you haven't, either. What I found does move the needle are: 1) Changing how much I charge, including adding upsells, and 2) Getting new sources of traffic.
- noahc 14y ago
  Sounds like you havent either.... That's like telling the director of the FBI he knows nothing about law enforcement because he wont reveal facts that he has a professional obligation to not reveal.
femto 14y ago
Are you aware of anyone experimenting with the automatic evolution of web pages? For example, use genetic programming to periodically modify the page served up. Both pages are served up, A/B style, and the page which has the highest metric (eg. $ earned) survives. Repeat. It would be interesting to see what the results are after an extended period of time.
- Estragon 14y ago
  Not quite what you're asking about, but an interesting related innovation: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...
- zackzackzack 14y ago
  Having played with genetic algorithms before, that is a hugely interesting idea to me. You would need to take special care on the types of mutations used though. I don't think it is possible to do it well at scale, but it would be very impressive nonetheless.

adrianhoward 14y ago

My hypothesis is A/B testing can move the needle for light engagement, like "try a free trial". But pulling out a credit card requires a lot of motivation, and the 0.1% of your visitors that have this motivation are relatively unaffected by your design. That's not true in my experience. I've seen some big increases in the percentages that complete the checkout process through an iterative process of redesign/test for example. The motivation behind a user purchase can be very different in different situations. If Mary is purchasing a new gearbox for the car, and she's already done a comparison shop, then she's likely to finish the purchase as long as it's vaguely sane. If Bob is impulse buying that new laptop bag he saw on swissmiss then every single grain of friction between Bob and the final "purchase" button is going to make it more likely that he remembers he really needs that $80 for groceries this week. At some point you're going to reach a maximum of course and only see small or no increases - but, with my clients anyway, folk seem to be quite a long way from that in many instances :-)