A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are t

A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are two different (and often complementary) activities. Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.