Low traffic levels do not mean that you cannot do any meaningful testing. It means that you cannot reliably detect subtle effects in the conversion rate. Wherea

Low traffic levels do not mean that you cannot do any meaningful testing. It means that you cannot reliably detect subtle effects in the conversion rate. Whereas Google may be able to reliably detect a 0.005% effect in the conversion rate, a small startup may only be able to reliably detect a 60% effect in the conversion rate. A small startup should arguably not be chasing after small effects in the conversion rate anyway. Evan Miller has a great sample size calculator to illustrate this: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html In the example above, Google's MDE is 0.005% and the small startup's MDE is 60%. If the small startup's baseline conversion rate is 5%, then with 95% confidence and 80% statistical power, they can reliably detect a 60% effect (positive or negative) with 1,968 visits. It's not just the 60% increase that you should be mindful of. I bet you all small startups want to know if the change they made to their homepage decreases the conversion rate by 60% or more. With an a/b test, they can do that. Another point to keep in mind is that you can learn a lot from a statistical tie (not enough data to conclude there is a difference). In fact, no matter what your traffic levels are, most of your experiments will be a statistical tie. It's important to learn what doesn't work just as it's important to know what does work. This can really help you with prioritization. In the example above, the small startup can use the statistical tie to draw the conclusion that some tasks in their product roadmap will result in less than a 60% effect in the conversion rate and can be prioritized as such. But ultimately it will be quite difficult for the small startup to produce variations that have at least a 60% effect in the conversion rate, but it's not unheard of and again, you can learn from a statistical tie. Just because you have low levels of traffic doesn't mean you can't learn anything from a/b testing.