I think the point "blindly following A/B tests is bad" is valid, but I don't completely follow the path this article paves to lead you there. For part one, reg

I think the point "blindly following A/B tests is bad" is valid, but I don't completely follow the path this article paves to lead you there. For part one, regarding forecasting, you were A/B testing within the segment you were targeting, whether you realized you were targeting that segment or not. You saw improvements within this segment, but were further alienating a blind spot. Strict, purely statistical A/B testing isn't a great mechanism for testing out segmentation strategies, business models, etc. A/B testing isn't a strategy to get you more oranges, it's a tactic to get more juice from the oranges you already have. For the second part, regarding the map, I'm still not sure if this is a good candidate for A/B testing, or if the implementation of the A/B test really isolated enough variables to reach a business conclusion. I suspect that may be difficult. Critical portions of your user interface are most likely another area where A/B testing is a bad idea.