2 ms·

I think what he is saying is that running minor test on extremely small traffic isn't going to give significant results soon and I would agree with him. Though

• btilly 17y ago
  Define "minor test". I say that because the biggest bang for the buck that I've seen from A/B tests have tended to be really small changes. Take out an unneeded form element. Take out the verbose explanations of questions so the form looked less intimidating. Choose an email subject line that attracted the right group of people for that email. Most developers would think of those as minor changes. None took more than a couple of hours to code. But I've seen those changes result in 10-40% lifts to the bottom line. That's not a minor change to the business as a whole. More importantly the figures that were quoted suggest about 800 conversions/day. The patience to let tests run 2-4 weeks can result in measuring that level of improvement. And after you have a few wins of that size, you start a virtuous cycle where every improvement makes it easier to raise traffic volume, which makes it easier to run A/B tests, which makes it easier to find further improvements.