A/B Testing is a manual hill-climbing algorithm, and we can't see the hill. Imagine a vast 2d plane that represents all the possible combinations of your websit

A/B Testing is a manual hill-climbing algorithm, and we can't see the hill. Imagine a vast 2d plane that represents all the possible combinations of your website - its buttons, colors, copy, layout. There are mountains and valleys that represent spots of high-signup and low-signup. You are at a certain point, and an A/B test will, say, move you North, at which point you can see what your altitude is. Is it higher? Lower? Unfortunately moving like this will trap you on a particular mountain, until someone comes along with a helicopter, saying - hey! I think I see a bigger mountain range over there! - and you try out a completely different design. Which you could certainly A/B test to see if the spot you landed on is actually higher than where you were (which if you're at the base of a huge mountain, may not be the case!). Of course, the combinations are multi-dimensional, and so are the fitness functions (signup, retention, word-of-mouth, etc). But I've found this useful to explain why A/B testing has problems; that is to say, it has specific uses, and so do UX designers.