3 ms·

Any particular change (like this redesign) might not move the needle, but there exists SUBSTANTIAL evidence that A/B testing can help the bottom line of the com

by patio11 15y ago

Any particular change (like this redesign) might not move the needle, but there exists SUBSTANTIAL evidence that A/B testing can help the bottom line of the company. Read e.g. any of the multiple times the 37 Signals blog covered how they got like 40% lifts to paid account signups by redoing a pricing or landing page. Do you find me credible on this topic? I'm legally, morally, and practically constrained regarding how much detail I can go into here, but if you trust me: I have recent experience which makes me disagree in the strongest possible way with the generalization of your hypothesis. A recent client CCs me on their weekly Optimizely status report. I've been evangelizing A/B testing for a few years. Every time I read one of those emails my mind gets blown again.

tomjen3 15y ago
But wouldn't there be big gains initially and then, as the easy fruit was plucked, the changes would give less and less and the successful tests would become fewer and fewer?
- lincolnq 15y ago
  Likely yes. But what matters is return on time in contrast to any of the other things you could be doing. "always work on the most important thing". Even small percentage point improvements in conversion go directly to the bottom line, which is far from obviously the case when fixing bugs or adding features. And if your bottom line is already pretty big, the marginal value of someone working on optimization is likely to continue to be high, regardless of how plucked the fruit seems to be.
jseims 15y ago
I'm sure others have achieved significant gains... I just haven't experienced them, and it sounds like you haven't, either. What I found does move the needle are: 1) Changing how much I charge, including adding upsells, and 2) Getting new sources of traffic.
- noahc 15y ago
  Sounds like you havent either.... That's like telling the director of the FBI he knows nothing about law enforcement because he wont reveal facts that he has a professional obligation to not reveal.
femto 15y ago
Are you aware of anyone experimenting with the automatic evolution of web pages? For example, use genetic programming to periodically modify the page served up. Both pages are served up, A/B style, and the page which has the highest metric (eg. $ earned) survives. Repeat. It would be interesting to see what the results are after an extended period of time.
- Estragon 15y ago
  Not quite what you're asking about, but an interesting related innovation: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...
- zackzackzack 15y ago
  Having played with genetic algorithms before, that is a hugely interesting idea to me. You would need to take special care on the types of mutations used though. I don't think it is possible to do it well at scale, but it would be very impressive nonetheless.

by patio11 15y ago

tomjen3 15y ago

But wouldn't there be big gains initially and then, as the easy fruit was plucked, the changes would give less and less and the successful tests would become fewer and fewer?

lincolnq 15y ago
Likely yes. But what matters is return on time in contrast to any of the other things you could be doing. "always work on the most important thing". Even small percentage point improvements in conversion go directly to the bottom line, which is far from obviously the case when fixing bugs or adding features. And if your bottom line is already pretty big, the marginal value of someone working on optimization is likely to continue to be high, regardless of how plucked the fruit seems to be.

jseims 15y ago

I'm sure others have achieved significant gains... I just haven't experienced them, and it sounds like you haven't, either. What I found does move the needle are: 1) Changing how much I charge, including adding upsells, and 2) Getting new sources of traffic.

noahc 15y ago
Sounds like you havent either.... That's like telling the director of the FBI he knows nothing about law enforcement because he wont reveal facts that he has a professional obligation to not reveal.

femto 15y ago

Are you aware of anyone experimenting with the automatic evolution of web pages? For example, use genetic programming to periodically modify the page served up. Both pages are served up, A/B style, and the page which has the highest metric (eg. $ earned) survives. Repeat. It would be interesting to see what the results are after an extended period of time.

Estragon 15y ago
Not quite what you're asking about, but an interesting related innovation: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...
zackzackzack 15y ago
Having played with genetic algorithms before, that is a hugely interesting idea to me. You would need to take special care on the types of mutations used though. I don't think it is possible to do it well at scale, but it would be very impressive nonetheless.