There’s a lot of false equivalence in this post. The message seems to be “if you want to build amazing new things, don’t A/B test.” This is vacuously true. You

There’s a lot of false equivalence in this post. The message seems to be “if you want to build amazing new things, don’t A/B test.” This is vacuously true. You certainly won’t discover fusion, or make the next iCar through rote testing. What’s hidden though is that out of the thousands of companies that will try, only a few will succeed. The rest will burn through millions and have nothing to show for it. A/B tests are done because they work in the short term, providing runway for the long term. It’s true that many companies abuse these tests and heavily lean on incremental improvements. Management can become complacent, spineless and rudderless, thereby refusing to make hard choices that go against the data. But, it keeps volatility down and appeases the overlords holding shares and vested interest in the company. It does make wonder though how, despite being so secretive, Apple got a lot of things right with the iPhone and iPod. So whatever their process was for internal focus groups is definitely worth exploring. We’ve all heard how Jobs was insanely attentive to details, but I still wonder where the overall cohesion of design came from. No one person could have possibly gotten all of that right.