4 ms·
The mechanics of A/B testing math make implementing this a much less attractive proposition than you'd think it would be. I'm going to get handwavy but it will
by patio11 15y ago
The mechanics of A/B testing math make implementing this a much less attractive proposition than you'd think it would be. I'm going to get handwavy but it will suffice for the moment: the amount of traffic you need to get statistical significance between N options grows at like O(N^2). How much of the color-space do you want to automatically converge over?
Most of my clients would struggle getting results at sufficient velocity with 10-way multivariate testing to say nothing of zomg-way multivariate testing.
Totally separate problem: Button color is far from the most salient thing you could be testing.
- sachinag 15y agoAnd if you're asking "what is?", the answer is probably your copy.
- deleted 15y ago[deleted]
- scottkrager 15y agoI've stopped doing minor multivariate testing in favor of A/B testing only major changes to a page. It's just too tempting to play around with color changes all day instead of building my business.
- krsgoss 15y agoYup. We wasted a considerable amount of time going for silly optimizations like this when we should of been trying changing the entire copy and layout. It felt like we were doing something when in fact we were just spinning our wheels.
- metadata 15y agoMe too. Major changes tend to have higher impact and provide more/faster learning. What exactly do I learn if I find that one color gives me 0.4% more conversions?