3 ms·
I apologize in advance if any of this is repetitive with things I have said before: 1) There is a difference between what you and I think are really stupid li
by patio11 17y ago
I apologize in advance if any of this is repetitive with things I have said before:
1) There is a difference between what you and I think are really stupid little things and what reality thinks are really stupid little things. For example, I think it is likely that you can (with a suitable A/B testing framework) change a call to action on your site in under five minutes of coding, counting the time to redeploy. Many engineers would consider that a stupid little change. Empirically, exactly that test has resulted in double-digit improvements before. So totally do that.
2) An A/B test which leads to the result "Not enough data to tell" doesn't convey zero information. It conveys an important bit of information: "Well, neither of these two alternatives lit a fire in the hearts of my users." That should give you the ability to recalibrate your efforts in the future.
3) With the context that I run a very part-time business: for the last couple of years I aimed at doing four A/B tests a month and getting a 5% increase out of one of them. If the other three end without significantly significant results, oh well. (See http://www.bingocardcreator.com/abingo/results http://www.bingocardcreator.com/abingo/results -- one in four isn't too far off what I actually get. Nota bene that is a convenience sampling of results I've actually accumulated, not all of them.) Of more applicability to you: you have to accept that A/B testing is a process which converges on awesome rather than a button which is onclick: deliverTheAwesome(). Over time, I promise you, it really does work. (It almost can't not work.) However, individual tests will frequently return null result or tell you that the new code you just spent time writing is a waste.
4) I try to avoid doing tests that take a week to set up unless I have a good reason to suspect it is going to seriously move the needle, because for the same amount of work you can throw a lot of smaller things at the wall and see what sticks. For example, I've put off an A/B test that I absolutely have to do until I am gainfully unemployed just because of the implementation difficulty. (A: current site selling online and downloadable software. B: what are you talking about, thee is only an online version.) Especially if you're new to A/B testing and don't have a good example that you can point to yourself and say "Bemmu, you feel like quitting, but REMEMBER THIS?! Oh good golly that worked out right. Test on!", test some stuff which takes 10 minutes to bang out alternatives for. Conversion buttons, calls to action, positioning of buttons, that sort of thing.
5) "Things that really move the needle" and "things that take a lot of time to implement" do not relate to each other. In fact, they sometimes have so little relation it is hilarious. Ask me how many weeks I have burned in developing features no one cares about versus how many minutes it took to change five words on my purchasing page. ("Buy a single copy via ..." -> "Get instant access to ...") Sure, go ahead and test the button designs. Let reality tell you what matters rather than thinking you have a good idea. I haven't the furthest clue.
- paraschopra 17y agohttp://www.bingocardcreator.com/abingo/results http://www.bingocardcreator.com/abingo/results Great, I never realized you had this page! A similar page for Visual Website Optimizer exists in the demo http://dev.visualwebsiteoptimizer.com/?demo=1 http://dev.visualwebsiteoptimizer.com/?demo=1
- bemmu 17y agoSo I wrote about the "not enough data to tell" experiment http://bemmu.posterous.com/12493093 http://bemmu.posterous.com/12493093