4 ms·
I wonder if the conclusion about A/B testing would have been the same if the tenuously-related lead-in anecdote had been about medicine ("Although randomized dr
by patio11 16y ago
I wonder if the conclusion about A/B testing would have been the same if the tenuously-related lead-in anecdote had been about medicine ("Although randomized drug trials might have succeeded in saving billions of lives, it didn't ring true to patients...")
You can ask users about their satisfaction. Do it. If you do A/B testing well, after testing it goes up. I'm sorry if it doesn't provide spiritual fulfillment to Jeff Atwood or product visionaries who hate the idea that something so simple so often trumps their vision, but surely you can work through that after you see the benefits available. A/B testing let me help over 100,000 extra kids learn to read this year -- that is pretty freaking spiritually compelling.
- tezza 16y agoHi Patrick... I'd just wonder if it would be safe to say that A/B testing is a good tool..., but should be used in conjunction with other tools such as knowing-your-customer, intuition, expertise etc. I'm not in a space where I can A/B test anything but from the outside it seems as if A/B testing can be mis-used as a replacement for traditional methods and can be obsessed about because it IS measurable.
- tel 16y agoThe local minimum point is correct I think. Finding the true maximum of a highly multidimensional quantity is fiendishly difficult. If A/B testing were sufficient then so many human and universal problems would vanish. So I totally agree with the article: you can polish with A/B, but unless you're pretty happy with the neighborhood you're in it's not the only tool in your belt.
- samd 16y agoA/B testing is great, but your data is severely limited. How do you know that sticking with the vision doesn't pay off in the long run? Unless you've been running A/B tests for years with doppleganger-Patrick who eschews the cold, mechanical methods of A/B testing, writes blog posts about how important gut-feeling is in web design, and made every page of BCC have yellow text on a green background because dammit he likes John Deere, how could you know that you're better off now? We know a lot less than we think we do, and running some limited tests doesn't change the fact that most of the time we're just guessing.
- AmberShah 16y agoThe problem I have with valuing A/B testing over the visionary is that A/B testing is a tool, not a purpose. The lean startup movement is everywhere and I keep seeing newbies coming onto the scene, reading a few articles and going "I don't really have an idea or a passion, but I'm going to A/B test my way to $1M business all by myself. But wait... how do I start?" It takes vision to decide what business to start and passion to follow it through and creativity to figure out what to try next. A/B tests -are- useful and they do have a prominent place in a business but please let's not pretend that's all it takes to build and grow a successful business. Look around at the founders of both VC backed and bootstrapped successful businesses and the common threads are much more about passion and persistence than the ability to A/B test.