So true about the danger of focusing on short term gains. I think that it is a problem of picking the wrong things to optimize and then sampling on the wrong e

So true about the danger of focusing on short term gains. I think that it is a problem of picking the wrong things to optimize and then sampling on the wrong entities. Conversion rate is often the most commonly used target, but, it is short sighted. In the big picture a company is really just trying to maximize profit. Note that you can't use traditional A/B testing tools on a value like total profit. You need an average if you are going to run an A/B test. For conversion rates your average is conversions per user. In this case you sample on the denominator -- the users. What do you sample on if you are testing total profit per quarter? This is the ultimate conundrum for websites who use A/B testing. They don't understand how they might actually get more new/returning users by simply improving the user experience. Instead, they run marketing campaigns to drive growth and then squeeze every dollar out of the customers when they arrive. This is not a good long term strategy. I have reviewed hundreds of A/B tests over the years. The only ones that I think succeeded were the ones that focused on improving the users experience.