2 ms·
My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes. That is
by btilly 17y ago
My takeaway from this is that it isn't worthwhile to do very small tests, but it is possible to get reliable results when doing fewer, bigger changes.
That is exactly wrong. You don't want to run a few big tests.
Instead run multiple minor tests at the same time. Make your tests be small changes - an email subject line, the size of a button, the help text you're using, etc. And test them in parallel to let you answer questions more quickly.
This strategy is not perfect because there may be interaction effects between your different tests. With larger traffic volumes you can address that problem. With smaller ones you need to hope that interactions are not significant. Usually interactions are not that significant, so you'll get away with the simple strategy.
What sorts of things should you test? Here are some ideas. Button size, page layout, help text, adding/removing form questions, email text, email subject lines, landing pages, registration pages, and so on and so forth. Interestingly the overall aesthetics of your site can be tested but seldom show a result. You tend to get payoffs from stuff that directly affects user interactions at points where users are not very motivated to continue with your site.
If you're going to be at the SXSW conference next month, the CEO of Freshbooks has a panel on A/B testing on Sunday that will give you some great concrete ideas of what has worked for a variety of companies. I highly recommend it. (Disclaimer, I'm on said panel.)
- paraschopra 17y agoI think what he is saying is that running minor test on extremely small traffic isn't going to give significant results soon and I would agree with him. Though I agree with you that doing minor tests is a better strategy given you have enough traffic for that.
- btilly 17y agoDefine "minor test". I say that because the biggest bang for the buck that I've seen from A/B tests have tended to be really small changes. Take out an unneeded form element. Take out the verbose explanations of questions so the form looked less intimidating. Choose an email subject line that attracted the right group of people for that email. Most developers would think of those as minor changes. None took more than a couple of hours to code. But I've seen those changes result in 10-40% lifts to the bottom line. That's not a minor change to the business as a whole. More importantly the figures that were quoted suggest about 800 conversions/day. The patience to let tests run 2-4 weeks can result in measuring that level of improvement. And after you have a few wins of that size, you start a virtuous cycle where every improvement makes it easier to raise traffic volume, which makes it easier to run A/B tests, which makes it easier to find further improvements.