4 ms·

I think Atwood misses the point a bit here: A/B testing is not destined to find shallow local maxima; one can also test larger, more significant differences. A

I think Atwood misses the point a bit here: A/B testing is not destined to find shallow local maxima; one can also test larger, more significant differences. And, ultimately, Phil (in Groundhog Day) discovers that.

paraschopra 16y ago
Right, there had been similar arguments by Rand of SEOMoz and Jason of Smartbear. A/B testing doesn't dictate what you are testing. It is an instrument -- ultimately you have to plan the experiment. A lot of people confuse that testing is just limited to doing red v/s green button color test. But what prevents you to converting your site to flash app and then A/B testing it. What does it have to do with creativity? Though all such articles are good at at least generating awareness about A/B testing.
mechanical_fish 16y ago
The problem, of course, is that you can only afford a certain number of big experiments. Life is finite. And big experiments can be hard to A/B test against your mature product. This is what that whole "chasm" thing is about: the audience for new things is different from the audience for old things; present a radical new thing to someone who isn't an early adopter and it will get low marks. The automobile had really bad A/B results against the horse, for most audiences in the late 1800s. It is hard work to attract a new audience, when you have the alternative of incrementally improving the experience of the old audience. And, of course, the more time you've invested in building your radically different idea, the more crushed you will be by bad A/B data. A/B testing of tiny differences that can be toggled in five minutes is the least emotionally painful form of A/B, so no wonder it is so much more popular, to the extent that Atwood thinks it represents the entire field. Atwood's metaphor is still perfect, however. The movie Groundhog Day covers all of this. It's an astonishing work of art.
deleted 16y ago
[deleted]

paraschopra 16y ago

Right, there had been similar arguments by Rand of SEOMoz and Jason of Smartbear. A/B testing doesn't dictate what you are testing. It is an instrument -- ultimately you have to plan the experiment. A lot of people confuse that testing is just limited to doing red v/s green button color test. But what prevents you to converting your site to flash app and then A/B testing it. What does it have to do with creativity? Though all such articles are good at at least generating awareness about A/B testing.

mechanical_fish 16y ago

The problem, of course, is that you can only afford a certain number of big experiments. Life is finite. And big experiments can be hard to A/B test against your mature product. This is what that whole "chasm" thing is about: the audience for new things is different from the audience for old things; present a radical new thing to someone who isn't an early adopter and it will get low marks. The automobile had really bad A/B results against the horse, for most audiences in the late 1800s. It is hard work to attract a new audience, when you have the alternative of incrementally improving the experience of the old audience. And, of course, the more time you've invested in building your radically different idea, the more crushed you will be by bad A/B data. A/B testing of tiny differences that can be toggled in five minutes is the least emotionally painful form of A/B, so no wonder it is so much more popular, to the extent that Atwood thinks it represents the entire field. Atwood's metaphor is still perfect, however. The movie Groundhog Day covers all of this. It's an astonishing work of art.

deleted 16y ago

[deleted]